Nov 02 22:45:13.197458 ip-10-0-135-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 22:45:13.197469 ip-10-0-135-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 22:45:13.197475 ip-10-0-135-179 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 22:45:13.197711 ip-10-0-135-179 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 22:45:23.447270 ip-10-0-135-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 22:45:23.447289 ip-10-0-135-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 84bb8911e988481984e384ae636cac68 -- Nov 02 22:47:24.444336 ip-10-0-135-179 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:47:24.799318 ip-10-0-135-179 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:24.799318 ip-10-0-135-179 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:47:24.799318 ip-10-0-135-179 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:24.799318 ip-10-0-135-179 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:47:24.805848 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805762 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.805848 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805846 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.805848 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805850 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.805848 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805853 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805856 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805859 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805862 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805864 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805872 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805875 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805878 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805883 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805886 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805891 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805893 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805896 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805898 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805901 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805903 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805906 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805908 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.805968 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805910 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805913 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805915 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805918 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805920 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805923 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805927 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805930 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805932 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805936 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805938 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805941 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805943 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805947 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805950 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805953 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805955 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805958 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805960 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805963 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.806445 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805966 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805968 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805971 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805974 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805977 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805980 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805982 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805985 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805987 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805990 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805992 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805994 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805997 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.805999 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806002 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806019 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806024 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806028 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806032 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.806907 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806034 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806037 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806039 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806047 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806049 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806052 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806055 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806058 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806060 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806064 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806066 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806069 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806071 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806074 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806076 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806079 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806081 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806084 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806086 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806089 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.807366 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806091 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806094 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806096 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806099 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806101 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806103 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806106 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806108 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806111 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806113 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806117 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806119 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806122 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806127 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806131 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806137 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806140 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806143 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.807843 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806146 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806149 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806152 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806154 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806157 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806917 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806924 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806927 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806929 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806932 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806935 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806938 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806940 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806942 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806945 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806948 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806951 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806953 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806956 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.808304 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806958 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806961 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806964 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806966 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806969 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806972 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806974 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806977 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806979 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806983 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806987 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806991 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806994 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806997 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.806999 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807002 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807020 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807023 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807026 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.808746 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807028 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807030 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807033 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807036 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807038 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807041 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807043 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807046 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807049 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807051 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807054 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807057 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807059 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807062 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807065 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807068 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807070 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807073 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807075 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807078 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.809218 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807081 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807083 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807086 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807088 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807091 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807097 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807099 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807102 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807105 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807107 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807110 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807113 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807115 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807118 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807120 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807123 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807125 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807128 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807130 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807134 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.809687 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807137 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807141 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807144 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807147 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807149 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807152 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807154 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807157 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807159 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807162 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807165 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807167 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807170 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807173 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807175 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807178 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807181 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807184 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807187 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.810172 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807190 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807194 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807196 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807199 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807201 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807203 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807206 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807208 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807211 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807213 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807216 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807218 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807220 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807223 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807730 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807736 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807739 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807742 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807745 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807747 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.810640 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807751 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807753 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807758 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807761 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807764 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807766 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807769 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807771 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807774 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807777 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807779 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807783 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807785 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807788 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807796 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807798 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807801 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.811116 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807803 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807806 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807808 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807811 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807813 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807816 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807818 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807822 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807826 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807829 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807832 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807835 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807837 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807840 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807843 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807846 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807849 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807854 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.811564 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807857 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807860 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807862 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807865 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807873 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807875 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807878 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807885 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807888 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807893 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807895 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807898 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807900 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807903 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.812059 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807905 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807907 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807910 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807913 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807915 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807918 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807920 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807923 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807925 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807928 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807931 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807933 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807936 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807938 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807940 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807943 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807946 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807948 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807950 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807953 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.812497 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807956 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807959 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807961 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807964 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807968 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807972 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807974 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807977 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807979 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807982 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807987 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807989 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807992 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807994 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807997 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.807999 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808002 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808019 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.813004 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808022 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808024 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808027 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.808029 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808927 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808940 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808948 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808952 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808959 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808963 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808967 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808976 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808979 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808982 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808986 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808989 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808992 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808995 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.808999 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809002 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809018 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809021 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809025 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809028 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:47:24.813459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809031 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809034 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809037 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809041 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809044 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809047 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809051 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809054 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809057 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809060 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809063 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809066 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809070 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809073 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809076 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809079 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809083 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809086 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809091 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809094 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809097 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809100 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809103 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809108 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:47:24.814004 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809111 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809114 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809117 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809121 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809123 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809126 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809129 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809132 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809135 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809138 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809142 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809145 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809148 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809151 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809154 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809157 2577 flags.go:64] FLAG: --help="false" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809160 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809163 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809166 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809169 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809172 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809176 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809178 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:47:24.814592 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809181 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809185 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809188 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809191 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809194 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809196 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809200 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809202 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809205 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809209 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809212 2577 flags.go:64] FLAG: --lock-file="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809214 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809217 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809222 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809228 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809231 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809235 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809238 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809241 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809244 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809247 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809250 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809254 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809257 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809261 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 22:47:24.815150 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809264 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809267 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809270 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809273 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809276 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809279 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809282 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809289 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809294 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809297 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809300 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809303 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809306 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809309 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809312 2577 flags.go:64] FLAG: --port="10250" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809314 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809317 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-066f3b221f3df8e08" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809321 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809324 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809327 2577 flags.go:64] FLAG: --register-node="true" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809331 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809335 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809338 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809341 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809343 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809347 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:47:24.815754 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809350 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809354 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809356 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809359 2577 flags.go:64] FLAG: --runonce="false" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809362 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809365 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809369 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809371 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809374 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809377 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809381 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809384 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809387 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809390 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809394 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809398 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809401 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809404 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809407 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809412 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809415 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809418 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809421 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809424 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809427 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:47:24.816787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809430 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809434 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809437 2577 flags.go:64] FLAG: --v="2" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809442 2577 flags.go:64] FLAG: --version="false" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809446 2577 flags.go:64] FLAG: --vmodule="" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809450 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809453 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809613 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809617 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809620 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809622 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809625 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809628 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809630 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809633 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809636 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809638 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809641 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809643 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809645 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809648 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.817495 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809655 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809657 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809660 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809662 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809665 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809667 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809670 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809673 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809675 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809677 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809681 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809684 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809687 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809690 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809692 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809695 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809698 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809701 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809705 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.818107 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809708 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809712 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809718 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809721 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809725 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809729 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809732 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809735 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809737 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809740 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809743 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809746 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809748 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809753 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809756 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809758 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809761 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.818566 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809763 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809765 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809768 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809771 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809773 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809778 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809781 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809783 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809786 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809788 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809791 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809793 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809796 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809798 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809801 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809803 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809805 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809808 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809810 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.819015 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809813 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809815 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809818 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809820 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809823 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809825 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809828 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809830 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809833 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809835 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809838 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809840 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809842 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809845 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809847 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809850 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809852 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809855 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809858 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809861 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.819483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809863 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809866 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809868 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809871 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809873 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809876 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809878 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809881 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809883 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809886 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809888 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809891 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809895 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.809922 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809966 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.819983 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809970 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809973 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809976 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809978 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809981 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809983 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809986 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809988 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809991 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809993 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809996 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.809998 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810000 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810003 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810018 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810023 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810026 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810039 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810042 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.820387 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810044 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810047 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810049 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810052 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810054 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810057 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810060 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810062 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810065 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810067 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810070 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810072 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810074 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810077 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810079 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810082 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810085 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810087 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810090 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.820849 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810092 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810095 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810097 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810099 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810102 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810104 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810107 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810109 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810112 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810116 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810119 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810122 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810124 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810127 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810130 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810132 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810134 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810137 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810139 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810142 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.821315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810144 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810147 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810149 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810152 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810154 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810164 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810167 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810169 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810174 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810177 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810179 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810182 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810185 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810187 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810190 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810192 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810194 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810197 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.821784 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810202 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810206 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810209 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810211 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810214 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810217 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810220 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810222 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810225 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810227 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810230 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810233 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810235 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810238 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810240 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810243 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810245 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810248 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810252 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.822315 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810255 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810258 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810262 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810265 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810268 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810272 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810275 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.810278 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.818987 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.819028 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.819814 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.819824 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819915 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819920 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819923 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819926 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819930 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819932 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819935 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819938 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.822759 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819940 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819943 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819945 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819948 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819950 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819953 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819955 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819958 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819960 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819963 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819965 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819968 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819970 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819974 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819979 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819983 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819986 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819989 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819992 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.823238 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819995 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.819998 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820001 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820004 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820024 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820026 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820029 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820032 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820034 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820037 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820039 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820042 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820045 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820047 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820050 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820052 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820055 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820057 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820060 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.823675 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820063 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820066 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820068 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820071 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820073 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820076 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820079 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820082 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820084 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820086 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820089 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820092 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820094 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820097 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820099 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820103 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820105 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820108 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820110 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.824139 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820114 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820118 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820121 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820124 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820127 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820129 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820131 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820134 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820136 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820139 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820141 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820144 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820146 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820149 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820152 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820155 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820157 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820160 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820162 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820165 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.824573 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820167 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820170 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820172 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820175 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820178 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820181 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820183 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820186 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820189 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820192 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820194 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820197 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820199 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820202 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820205 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820207 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820210 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820212 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820215 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820217 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.825058 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820220 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.820248 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820401 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820406 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820409 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820411 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820414 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820417 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820420 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820423 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820426 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820428 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820431 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820433 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820436 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820438 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.825515 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820441 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820443 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820447 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820452 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820456 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820459 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820461 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820464 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820466 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820469 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820471 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820474 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820476 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820478 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820481 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820484 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820486 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820489 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820491 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820493 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.825887 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820496 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820499 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820501 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820503 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820509 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820511 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820514 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820516 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820519 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820521 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820524 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820526 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820529 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820531 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820534 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820536 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820539 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820542 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820544 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.826483 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820547 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820549 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820552 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820554 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820557 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820559 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820562 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820564 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820567 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820569 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820572 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820574 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820577 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820579 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820582 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820584 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820587 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820589 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820592 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820595 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.826913 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820597 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820599 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820602 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820605 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820607 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820609 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820612 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820614 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820617 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820620 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820622 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820625 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820627 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820629 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820632 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820635 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820637 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820640 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820642 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.827392 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820646 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820649 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820655 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820658 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820661 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820664 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820667 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820669 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820672 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820674 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820677 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820679 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820682 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.820705 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820738 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:47:24.827835 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820742 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820744 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820747 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820749 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820754 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820757 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820759 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820762 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820765 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820767 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820770 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820772 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820775 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820778 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820780 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820783 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820785 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820788 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820797 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:47:24.828227 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820799 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820803 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820807 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820810 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820816 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820819 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820822 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820829 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820832 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820835 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820837 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820840 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820842 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820845 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820847 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820851 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820854 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820856 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:47:24.828699 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820859 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820862 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820865 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820867 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820870 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820872 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820875 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820878 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820880 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820883 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820885 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820888 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820890 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820892 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820895 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820897 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820900 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820904 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:47:24.829175 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820907 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820909 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820912 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820914 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820917 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820920 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820922 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820925 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820927 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820930 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820933 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820936 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820938 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820941 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820944 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820946 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820949 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820951 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820954 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820957 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:47:24.829606 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820959 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820961 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820964 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820967 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820969 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820972 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820974 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820976 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820979 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820981 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820984 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820987 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820990 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820993 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820995 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.820998 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821000 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821003 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821022 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:47:24.830079 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821027 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821031 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821034 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821040 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821042 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821045 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821047 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:24.821050 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.821112 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.823589 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.824511 2577 server.go:1014] "Starting client certificate rotation" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.824605 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.824643 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.826599 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:24.830567 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.827850 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:47:24.839620 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.839598 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:47:24.843828 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.843811 2577 log.go:25] "Validated CRI v1 image API" Nov 02 22:47:24.845437 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.845417 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:47:24.849339 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.849313 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ccb82836-6794-421b-8035-61b033247959:/dev/nvme0n1p4 cea4f880-4b4c-474e-9a93-165bf2c7a256:/dev/nvme0n1p3] Nov 02 22:47:24.849413 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.849338 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:47:24.855178 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.855053 2577 manager.go:217] Machine: {Timestamp:2026-11-02 22:47:24.853245971 +0000 UTC m=+0.310361082 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108271 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225e6d36ec6733b43309850ba73356 SystemUUID:ec225e6d-36ec-6733-b433-09850ba73356 BootID:84bb8911-e988-4819-84e3-84ae636cac68 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:47:20:e7:6a:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:47:20:e7:6a:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:f6:c4:97:77:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:47:24.855811 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.855797 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:47:24.855923 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.855906 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:47:24.857910 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.857880 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:47:24.858099 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.857912 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-179.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:47:24.858144 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.858102 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:47:24.858144 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.858111 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:47:24.858200 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.858150 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:24.858200 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.858167 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:47:24.859398 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.859388 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:47:24.863793 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.863781 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:47:24.863904 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.863895 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:47:24.864707 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.864697 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:47:24.864753 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.864709 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:47:24.864753 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.864723 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:47:24.864753 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.864731 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:47:24.864753 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.864740 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:47:24.865123 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.865109 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:24.867422 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.867396 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:47:24.874796 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.874783 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:47:24.875741 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875729 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:47:24.875785 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875760 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:47:24.875913 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875903 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:47:24.875940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875919 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:47:24.875940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875925 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:47:24.875940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875931 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:47:24.875940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875936 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:47:24.875940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.875942 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:47:24.876693 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876684 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:47:24.876721 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876694 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:47:24.876721 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876701 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:47:24.876721 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876709 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:47:24.876789 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876724 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:47:24.876789 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.876732 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:47:24.877494 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.877486 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:47:24.877535 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.877495 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:47:24.881488 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.881473 2577 server.go:1263] "Started kubelet" Nov 02 22:47:24.881558 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.881521 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:47:24.881655 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.881616 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:47:24.881693 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.881682 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:47:24.882970 ip-10-0-135-179 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:47:24.883447 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.883159 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:47:24.887175 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.887157 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:47:24.889196 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:24.889177 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:47:24.890511 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.890498 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:24.891127 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.891111 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:47:24.891757 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:24.891316 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" Nov 02 22:47:24.892025 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.891989 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:47:24.892025 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.891991 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:47:24.892155 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892036 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:47:24.892155 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892135 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:47:24.892155 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892140 2577 factory.go:153] Registering CRI-O factory Nov 02 22:47:24.892155 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892154 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892143 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892202 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892213 2577 factory.go:55] Registering systemd factory Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892221 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892234 2577 factory.go:103] Registering Raw factory Nov 02 22:47:24.893118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.892251 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 22:47:24.893392 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.893330 2577 manager.go:319] Starting recovery of all containers Nov 02 22:47:24.904657 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.904580 2577 manager.go:324] Recovery completed Nov 02 22:47:24.908474 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.908459 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:24.910473 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.910449 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:24.910537 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.910485 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:24.910537 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.910496 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:24.910998 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.910985 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:47:24.910998 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.910995 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:47:24.911147 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.911000 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:24.911147 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.911025 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:47:24.911744 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.911728 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:24.911816 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.911760 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:24.911816 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.911774 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:24.913174 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.913162 2577 policy_none.go:50] "Start" Nov 02 22:47:24.913225 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.913178 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:47:24.913225 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.913187 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:47:24.915180 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.915170 2577 policy_none.go:44] "Start" Nov 02 22:47:24.951605 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.951572 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:47:24.953898 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.953883 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 22:47:24.954003 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:24.953915 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:47:24.954003 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.953924 2577 server.go:87] "Starting device plugin registration server" Nov 02 22:47:24.954168 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.954154 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:47:24.954220 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.954168 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:47:24.954290 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.954243 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:47:24.954350 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.954330 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:47:24.954350 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:24.954337 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:47:24.955051 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:24.954849 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:47:24.955132 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:24.955086 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" Nov 02 22:47:25.025348 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.025323 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-179.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:47:25.025348 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.025325 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-179.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:47:25.027246 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.025388 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.us-west-1.compute.internal.18e45bf19d1a0414 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:24.881273876 +0000 UTC m=+0.338388980,LastTimestamp:2026-11-02 22:47:24.881273876 +0000 UTC m=+0.338388980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.us-west-1.compute.internal,}" Nov 02 22:47:25.033188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.033162 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:47:25.034506 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.034485 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:47:25.034506 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.034507 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:47:25.034655 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.034532 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:47:25.034655 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.034566 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:47:25.054552 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.054509 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:25.055306 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.055287 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:25.055381 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.055321 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:25.055381 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.055332 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:25.055381 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.055353 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.110512 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.110492 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:25.110591 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.110529 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kf4z2" Nov 02 22:47:25.135486 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.135465 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal"] Nov 02 22:47:25.135564 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.135523 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:25.140468 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.140454 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:25.140549 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.140481 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:25.140549 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.140491 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:25.142694 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.142675 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:25.142832 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.142820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.142873 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.142851 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:25.143419 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143401 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:25.143492 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143405 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:25.143492 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143455 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:25.143492 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143467 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:25.143492 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143432 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:25.143620 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.143509 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:25.145479 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.145454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.145479 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.145480 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:47:25.146206 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.146192 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:47:25.146308 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.146214 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:47:25.146308 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.146223 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:47:25.174418 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.174402 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.177000 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.176982 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.184183 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.184166 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:25.185149 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.185135 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:25.195293 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.195271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.195375 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.195299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.195375 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.195318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.283730 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.283697 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:25.284183 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.284167 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:25.285118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.285102 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.285199 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.285124 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-179.us-west-1.compute.internal\": node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" Nov 02 22:47:25.291825 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.291813 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296085 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296140 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296140 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296135 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296214 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296214 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296166 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.296214 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.296180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.352797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.352745 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:25.352878 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.352842 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.409580 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.409555 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:47:25.477041 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.477000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.479830 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.479811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:47:25.824746 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.824671 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:47:25.825336 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.824792 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:25.825336 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.824819 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:25.825336 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.824848 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:47:25.825336 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:25.824863 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae6c114d507d041eea76c03a1ba35e05-cac881102500eba0.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-179.us-west-1.compute.internal.18e45bf19ed7f801\": read tcp 10.0.135.179:33426->77.113.138.105:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-179.us-west-1.compute.internal.18e45bf19ed7f801 default 4944 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-179.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 22:47:24 +0000 UTC,LastTimestamp:2026-11-02 22:47:25.055335519 +0000 UTC m=+0.512450622,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.us-west-1.compute.internal,}" Nov 02 22:47:25.865669 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.865646 2577 apiserver.go:52] "Watching apiserver" Nov 02 22:47:25.890942 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:25.890917 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:47:26.114551 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.114474 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:26 +0000 UTC" deadline="2028-06-28 07:26:59.565662278 +0000 UTC" Nov 02 22:47:26.114551 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.114504 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14480h39m33.451162961s" Nov 02 22:47:26.134846 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.134808 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:47:26.135160 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.135141 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-drs9r","openshift-multus/multus-cb9nm","openshift-cluster-node-tuning-operator/tuned-8fszs","openshift-dns/node-resolver-7jpbd","openshift-multus/network-metrics-daemon-g6zks","openshift-network-diagnostics/network-check-target-9zkr2","openshift-ovn-kubernetes/ovnkube-node-t92fq","kube-system/global-pull-secret-syncer-k9d98","kube-system/konnectivity-agent-792jn","kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-rrqq2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw","openshift-image-registry/node-ca-px8c4"] Nov 02 22:47:26.137477 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.137463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.139482 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.139463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.141397 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.141382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.143398 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.143383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.145417 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.145399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.145493 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.145474 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:26.147394 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.147376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:26.147486 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.147426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:26.149538 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.149522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.151684 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.151666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.151760 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.151725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:26.153815 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.153800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.155991 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.155977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.158075 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.158063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.160233 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.160220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.189824 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.189801 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.189905 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.189860 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.189967 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.189922 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:47:26.189967 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.189934 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:47:26.189967 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.189941 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:47:26.190289 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.190274 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:47:26.190854 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.190829 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:47:26.191041 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.191023 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:47:26.192737 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.192723 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.192797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.192760 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.192942 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.192925 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:47:26.193134 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.193117 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:47:26.194940 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.194923 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.194979 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.194932 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:47:26.194979 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.194963 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.201279 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rsjr\" (UniqueName: \"kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.201361 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.201361 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.201464 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.201464 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.201464 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201459 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:47:26.201601 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.201601 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7rg\" (UniqueName: \"kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.201601 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.201601 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201550 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:47:26.201601 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfjx\" (UniqueName: \"kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201632 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201682 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.201826 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201817 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201892 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.201980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tt5\" (UniqueName: \"kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklls\" (UniqueName: \"kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.202651 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hkl\" (UniqueName: \"kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:26.203201 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.202990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2nt\" (UniqueName: \"kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcz29\" (UniqueName: \"kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4phx\" (UniqueName: \"kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.203787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.204268 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.204268 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.204268 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.204268 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.204268 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.203577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.239369 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.239350 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:47:26.239775 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.239751 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:47:26.239861 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.239795 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:47:26.239861 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.239804 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.239954 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.239892 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:47:26.240631 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.240618 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:47:26.241118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.240640 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.241118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.240723 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.241118 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.240772 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.242627 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.242614 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:47:26.244445 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244431 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:47:26.244711 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244688 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:47:26.244711 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244700 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:47:26.244782 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244770 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:47:26.244782 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244777 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:47:26.244873 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.244836 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:47:26.251122 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.251102 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:47:26.251210 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.251177 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmgnk" Nov 02 22:47:26.304678 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklls\" (UniqueName: \"kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.304797 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304830 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304841 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.304946 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305092 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.305116 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.305192 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.805155977 +0000 UTC m=+2.262271096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305339 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hkl\" (UniqueName: \"kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305399 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305487 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.305603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2nt\" (UniqueName: \"kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305597 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305610 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz29\" (UniqueName: \"kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4phx\" (UniqueName: \"kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.306377 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306546 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.306885 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsjr\" (UniqueName: \"kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.307188 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307150 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.306686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307176 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.307232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.806945248 +0000 UTC m=+2.264060352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.305445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7rg\" (UniqueName: \"kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307409 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.308244 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308283 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.307579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfjx\" (UniqueName: \"kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308932 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.308984 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.309053 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309034 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309410 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309416 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.309502 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.309813 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309767 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.309978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tt5\" (UniqueName: \"kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310084 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310102 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310137 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.310489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310613 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.310933 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.310985 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311057 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311159 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.311212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.311463 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311243 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.311463 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311260 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.311463 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311336 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.311463 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311386 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.311463 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.311833 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.311807 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.313873 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.312595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.313873 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.312705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.314057 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.313949 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.314464 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.314445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.363394 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.363366 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:26.363394 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.363393 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:26.363566 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.363405 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:26.363566 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:26.363471 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:26.863457089 +0000 UTC m=+2.320572201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:26.365522 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.365449 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklls\" (UniqueName: \"kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.365812 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.365793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hkl\" (UniqueName: \"kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.366441 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.366406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2nt\" (UniqueName: \"kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.366538 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.366512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz29\" (UniqueName: \"kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.366538 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.366522 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4phx\" (UniqueName: \"kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:26.367357 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.367332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsjr\" (UniqueName: \"kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.368251 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.368226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfjx\" (UniqueName: \"kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.368449 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.368434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7rg\" (UniqueName: \"kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.370838 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.370818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tt5\" (UniqueName: \"kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.478616 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.478582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb9nm" Nov 02 22:47:26.483850 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.483826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:47:26.492657 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.492632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:47:26.507422 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.507398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:47:26.535176 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.535146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:26.537269 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.537245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:26.551787 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.551768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:47:26.568030 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.567996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:47:26.570082 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.570062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:47:26.613874 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:26.613831 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd1027324a8515ba7169fd6975c2633.slice/crio-7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7 WatchSource:0}: Error finding container 7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7: Status 404 returned error can't find the container with id 7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7 Nov 02 22:47:26.614170 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:26.614154 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bcc47f51109dce7950aafef82fe134.slice/crio-00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006 WatchSource:0}: Error finding container 00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006: Status 404 returned error can't find the container with id 00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006 Nov 02 22:47:26.618971 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:26.618957 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:47:28.036342 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.036318 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:28.164639 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.164608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:28.164814 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.164673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:28.164814 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.164778 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:28.164814 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.164790 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:28.164975 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.164837 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:29.164822013 +0000 UTC m=+3.271774191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:28.164975 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.164851 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:29.164845256 +0000 UTC m=+3.271797416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:28.265381 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.265296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:28.265885 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.265424 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:28.265885 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.265437 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:28.265885 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.265447 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:28.265885 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:28.265492 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:29.265478733 +0000 UTC m=+3.372430893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:28.389151 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.389094 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"b8bcc47f51109dce7950aafef82fe134","Type":"ContainerStarted","Data":"00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006"} Nov 02 22:47:28.390132 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.390110 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerStarted","Data":"7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7"} Nov 02 22:47:28.604227 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.604118 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:27 +0000 UTC" deadline="2028-04-12 18:09:19.41686232 +0000 UTC" Nov 02 22:47:28.604227 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.604148 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12643h21m50.812719434s" Nov 02 22:47:28.617840 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.617819 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:28.650166 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:28.650137 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:47:28.674333 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.674293 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf726490e_b32d_43ec_a286_750ec26f808d.slice/crio-82027de42ffe46bc2b7c0484275ac4c05cdd971aaeee4868d88551c134fdd7b1 WatchSource:0}: Error finding container 82027de42ffe46bc2b7c0484275ac4c05cdd971aaeee4868d88551c134fdd7b1: Status 404 returned error can't find the container with id 82027de42ffe46bc2b7c0484275ac4c05cdd971aaeee4868d88551c134fdd7b1 Nov 02 22:47:28.723496 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.723468 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76d6a1d_5b9b_4bbe_8848_f1400d29d16d.slice/crio-d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf WatchSource:0}: Error finding container d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf: Status 404 returned error can't find the container with id d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf Nov 02 22:47:28.743350 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.743325 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda686ab8e_a865_4f13_853b_3d43a3eef118.slice/crio-a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1 WatchSource:0}: Error finding container a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1: Status 404 returned error can't find the container with id a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1 Nov 02 22:47:28.746147 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.746123 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cb3950_1c7d_4855_a7b7_eaf1b6fa095c.slice/crio-ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62 WatchSource:0}: Error finding container ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62: Status 404 returned error can't find the container with id ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62 Nov 02 22:47:28.746792 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.746774 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c1fe43_01b8_4f60_a4da_97456b7ec024.slice/crio-9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d WatchSource:0}: Error finding container 9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d: Status 404 returned error can't find the container with id 9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d Nov 02 22:47:28.773881 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.773850 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555b9232_5b06_42ff_ae4f_72bf6968ef70.slice/crio-cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b WatchSource:0}: Error finding container cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b: Status 404 returned error can't find the container with id cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b Nov 02 22:47:28.781886 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.781854 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938e7ec8_af54_4d4f_b0c7_507a87ebb9e8.slice/crio-227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d WatchSource:0}: Error finding container 227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d: Status 404 returned error can't find the container with id 227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d Nov 02 22:47:28.796875 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.796852 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c23212_0707_4228_984b_9f8d412ad99c.slice/crio-9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e WatchSource:0}: Error finding container 9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e: Status 404 returned error can't find the container with id 9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e Nov 02 22:47:28.874769 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:28.874739 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a94f467_aab7_40b5_a713_356acd286516.slice/crio-5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736 WatchSource:0}: Error finding container 5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736: Status 404 returned error can't find the container with id 5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736 Nov 02 22:47:29.173501 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.173411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:29.173501 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.173486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:29.173670 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.173599 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:29.173670 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.173653 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:31.173636678 +0000 UTC m=+5.280588852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:29.173738 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.173716 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:29.173772 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.173743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:31.173731635 +0000 UTC m=+5.280683811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:29.274013 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.273981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:29.274608 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.274108 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:29.274608 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.274122 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:29.274608 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.274131 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:29.274608 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.274205 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:31.274190034 +0000 UTC m=+5.381142215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:29.385366 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.385336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:29.385537 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.385465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:29.385537 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.385475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:29.385537 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.385473 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:29.385698 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.385586 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:29.385698 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:29.385676 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:29.393477 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.393447 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrqq2" event={"ID":"b7c1fe43-01b8-4f60-a4da-97456b7ec024","Type":"ContainerStarted","Data":"9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d"} Nov 02 22:47:29.394627 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.394599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-792jn" event={"ID":"a686ab8e-a865-4f13-853b-3d43a3eef118","Type":"ContainerStarted","Data":"a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1"} Nov 02 22:47:29.396230 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.396210 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerStarted","Data":"d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf"} Nov 02 22:47:29.397304 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.397281 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8fszs" event={"ID":"f726490e-b32d-43ec-a286-750ec26f808d","Type":"ContainerStarted","Data":"82027de42ffe46bc2b7c0484275ac4c05cdd971aaeee4868d88551c134fdd7b1"} Nov 02 22:47:29.398537 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.398513 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-px8c4" event={"ID":"5a94f467-aab7-40b5-a713-356acd286516","Type":"ContainerStarted","Data":"5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736"} Nov 02 22:47:29.399770 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.399737 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb9nm" event={"ID":"f3c23212-0707-4228-984b-9f8d412ad99c","Type":"ContainerStarted","Data":"9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e"} Nov 02 22:47:29.400968 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.400938 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jpbd" event={"ID":"555b9232-5b06-42ff-ae4f-72bf6968ef70","Type":"ContainerStarted","Data":"cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b"} Nov 02 22:47:29.402028 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.402004 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62"} Nov 02 22:47:29.403183 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.403151 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" event={"ID":"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8","Type":"ContainerStarted","Data":"227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d"} Nov 02 22:47:29.605152 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.605042 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:27 +0000 UTC" deadline="2028-07-29 14:05:19.446701658 +0000 UTC" Nov 02 22:47:29.605152 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:29.605099 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15231h17m49.841606274s" Nov 02 22:47:31.194917 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.194841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:31.194917 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.194895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:31.195601 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.195014 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:31.195601 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.195069 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.19505176 +0000 UTC m=+9.302003933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:31.195601 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.195384 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:31.195601 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.195417 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.195408152 +0000 UTC m=+9.302360326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:31.295875 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.295833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:31.296057 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.295996 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:31.296057 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.296018 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:31.296057 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.296030 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:31.296235 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.296087 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:35.296067303 +0000 UTC m=+9.403019487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.386012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.386158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.386572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.386665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:31.386728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:31.386866 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:31.386804 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:32.436511 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:32.436243 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="3c673435694cd266834fa435aed9c85afcea1a30ceed75732b3a63c88f1cc820" exitCode=0 Nov 02 22:47:32.436511 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:32.436445 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"3c673435694cd266834fa435aed9c85afcea1a30ceed75732b3a63c88f1cc820"} Nov 02 22:47:32.442522 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:32.442013 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"b8bcc47f51109dce7950aafef82fe134","Type":"ContainerStarted","Data":"c4fc8d03a4d56851e998a2b151b68aaa4f2e7266e65055ce9f0fc90486bba111"} Nov 02 22:47:32.455083 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:32.454990 2577 generic.go:356] "Generic (PLEG): container finished" podID="3bd1027324a8515ba7169fd6975c2633" containerID="c20d8d51aab963c612d198855425998b9ec2b8e5626160e9e31471573e163b69" exitCode=0 Nov 02 22:47:32.455083 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:32.455035 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerDied","Data":"c20d8d51aab963c612d198855425998b9ec2b8e5626160e9e31471573e163b69"} Nov 02 22:47:33.385823 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.385791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:33.385994 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:33.385930 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:33.385994 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.385987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:33.386110 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:33.386061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:33.386110 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.386103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:33.386230 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:33.386193 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:33.464225 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.464161 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerStarted","Data":"47b42536e078b1a4dec9659c0d8a24fdac8d1924e7bd401d97ae4a962c31e3d4"} Nov 02 22:47:33.578045 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.577989 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" podStartSLOduration=7.577973244 podStartE2EDuration="7.577973244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:32.800278133 +0000 UTC m=+6.907230318" watchObservedRunningTime="2026-11-02 22:47:33.577973244 +0000 UTC m=+7.684925486" Nov 02 22:47:33.578226 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:33.578063 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" podStartSLOduration=7.578059425 podStartE2EDuration="7.578059425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:33.577370324 +0000 UTC m=+7.684322508" watchObservedRunningTime="2026-11-02 22:47:33.578059425 +0000 UTC m=+7.685011608" Nov 02 22:47:34.477857 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:34.477821 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-792jn" event={"ID":"a686ab8e-a865-4f13-853b-3d43a3eef118","Type":"ContainerStarted","Data":"8ae8e78c60d908175c6e92042076545e34e2d4e5a85abda167467858ee365e5a"} Nov 02 22:47:35.049116 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.049035 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:35.049775 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.049753 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:35.165872 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.165813 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-792jn" podStartSLOduration=4.5824841549999995 podStartE2EDuration="9.165795568s" totalImagesPullingTime="4.583311413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:34.594270156 +0000 UTC m=+8.701222339" watchObservedRunningTime="2026-11-02 22:47:35.165795568 +0000 UTC m=+9.272747753" Nov 02 22:47:35.229666 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.229592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:35.229978 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.229718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:35.229978 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.229868 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:35.229978 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.229932 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:43.229914387 +0000 UTC m=+17.336866561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:35.230091 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.229994 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:35.230091 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.230030 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:43.230018328 +0000 UTC m=+17.336970504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:35.331190 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.331130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:35.331371 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.331317 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:35.331371 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.331340 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:35.331371 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.331353 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:35.331526 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.331409 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:43.331391971 +0000 UTC m=+17.438344147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:35.385432 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.385403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:35.385608 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.385445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:35.385608 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.385413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:35.385608 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.385555 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:35.385758 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.385616 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:35.385758 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:35.385684 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:35.480988 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.480906 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" event={"ID":"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8","Type":"ContainerStarted","Data":"a2f845bdf9fc40075f86c0add069229d9d51f3da15641349d449045da1cb128b"} Nov 02 22:47:35.484217 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.484185 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-px8c4" event={"ID":"5a94f467-aab7-40b5-a713-356acd286516","Type":"ContainerStarted","Data":"91a9796d6ca4ccecc64f595c5224c89066e04700d005040a7a037d1e22f56010"} Nov 02 22:47:35.597980 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:35.597920 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-px8c4" podStartSLOduration=3.484875569 podStartE2EDuration="9.597899751s" totalImagesPullingTime="6.113024182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:35.597081329 +0000 UTC m=+9.704033549" watchObservedRunningTime="2026-11-02 22:47:35.597899751 +0000 UTC m=+9.704851936" Nov 02 22:47:36.488202 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:36.488064 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:47:37.142647 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.142618 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:47:37.309809 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.309701 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:47:37.142644451Z","UUID":"94da33e7-de96-48c5-967e-d54cfda87b7d","Handler":null,"Name":"","Endpoint":""} Nov 02 22:47:37.311371 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.311351 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:47:37.311518 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.311380 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.385730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:37.385859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.386278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:37.386382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.386441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:37.386548 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:37.386508 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:37.499121 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:37.499088 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" event={"ID":"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8","Type":"ContainerStarted","Data":"f204cf9afb53273e3e5826493cfa4f10c7304d8de854c50b8428775beab7652b"} Nov 02 22:47:38.503435 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.503392 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb9nm" event={"ID":"f3c23212-0707-4228-984b-9f8d412ad99c","Type":"ContainerStarted","Data":"7645917e457d21496c26194208b3c6eda4a4f2125d511facb31ed07de965f81b"} Nov 02 22:47:38.506220 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.506190 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jpbd" event={"ID":"555b9232-5b06-42ff-ae4f-72bf6968ef70","Type":"ContainerStarted","Data":"c5d40b55078d6f115c905d94ba887010bc06996d51782b1b4d19e8479d6e0de3"} Nov 02 22:47:38.508686 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.508663 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="57e5981938fb44fd27151fe3758273b8b23321b6774a2cf1c67b0e668e7f639d" exitCode=0 Nov 02 22:47:38.508804 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.508706 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"57e5981938fb44fd27151fe3758273b8b23321b6774a2cf1c67b0e668e7f639d"} Nov 02 22:47:38.743129 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.743085 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cb9nm" podStartSLOduration=4.148258507 podStartE2EDuration="12.743072404s" totalImagesPullingTime="8.594813897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:38.62210818 +0000 UTC m=+12.729060362" watchObservedRunningTime="2026-11-02 22:47:38.743072404 +0000 UTC m=+12.850024587" Nov 02 22:47:38.857489 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:38.856966 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7jpbd" podStartSLOduration=4.203266793 podStartE2EDuration="12.856949166s" totalImagesPullingTime="8.653682373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:38.856419792 +0000 UTC m=+12.963372106" watchObservedRunningTime="2026-11-02 22:47:38.856949166 +0000 UTC m=+12.963901353" Nov 02 22:47:39.385102 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.384934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:39.385264 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.384978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:39.385264 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:39.385188 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:39.385264 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.384985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:39.385369 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:39.385265 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:39.385369 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:39.385325 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:39.513466 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.513390 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" event={"ID":"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8","Type":"ContainerStarted","Data":"fe4a44aa7c42920398310023b024772669254abe4d1607b0481e89ebe157bbe0"} Nov 02 22:47:39.515808 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.515777 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrqq2" event={"ID":"b7c1fe43-01b8-4f60-a4da-97456b7ec024","Type":"ContainerStarted","Data":"b678ae0a02dfe3b6ff68207e8d036059c398b0c6c37a84428170f0047cc94394"} Nov 02 22:47:39.517857 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.517792 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8fszs" event={"ID":"f726490e-b32d-43ec-a286-750ec26f808d","Type":"ContainerStarted","Data":"cdd74b289d4fe5758bdfafb825660f148dfc1cc5e35195d64122c5bdf7a29d61"} Nov 02 22:47:39.631460 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.631394 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" podStartSLOduration=4.287365488 podStartE2EDuration="13.631376193s" totalImagesPullingTime="9.344010705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:39.631246164 +0000 UTC m=+13.738198348" watchObservedRunningTime="2026-11-02 22:47:39.631376193 +0000 UTC m=+13.738328378" Nov 02 22:47:39.745949 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.745897 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrqq2" podStartSLOduration=5.282045579 podStartE2EDuration="13.745869143s" totalImagesPullingTime="8.463823564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:39.745458165 +0000 UTC m=+13.852410348" watchObservedRunningTime="2026-11-02 22:47:39.745869143 +0000 UTC m=+13.852821326" Nov 02 22:47:39.861398 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:39.861350 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8fszs" podStartSLOduration=3.361865549 podStartE2EDuration="13.861336814s" totalImagesPullingTime="10.499471265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:39.861026999 +0000 UTC m=+13.967979185" watchObservedRunningTime="2026-11-02 22:47:39.861336814 +0000 UTC m=+13.968288996" Nov 02 22:47:40.522994 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:40.522957 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="aeeca20ae6fc636a8f0d2917fe60d23c2023a227a43350c1f2e3c73a4adb97a5" exitCode=0 Nov 02 22:47:40.523516 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:40.523094 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"aeeca20ae6fc636a8f0d2917fe60d23c2023a227a43350c1f2e3c73a4adb97a5"} Nov 02 22:47:40.568950 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:40.568920 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:40.569121 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:40.569042 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:47:40.569704 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:40.569682 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:47:41.385401 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:41.385366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:41.385566 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:41.385442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:41.385566 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:41.385468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:41.385566 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:41.385550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:41.385711 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:41.385629 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:41.385711 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:41.385664 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:42.529615 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:42.529585 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="c86245df72270d63d9f00827da287b8570f4d3668d9281d13c818381c321a44a" exitCode=0 Nov 02 22:47:42.530474 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:42.529634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"c86245df72270d63d9f00827da287b8570f4d3668d9281d13c818381c321a44a"} Nov 02 22:47:43.298370 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.298232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:43.298452 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.298368 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:43.298452 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.298409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:43.298452 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.298422 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret podName:1555d5cf-a68b-4168-bff4-da817535b17e nodeName:}" failed. No retries permitted until 2026-11-02 22:47:59.298407233 +0000 UTC m=+33.405359407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret") pod "global-pull-secret-syncer-k9d98" (UID: "1555d5cf-a68b-4168-bff4-da817535b17e") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:47:43.298540 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.298478 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:43.298540 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.298518 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:59.298507911 +0000 UTC m=+33.405460089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:47:43.385667 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.385639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:43.385772 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.385650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:43.385835 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.385783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:43.385835 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.385656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:43.385934 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.385884 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:43.386018 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.385993 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:43.401513 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.401481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:43.401722 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.401703 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:47:43.401801 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.401729 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:47:43.401801 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.401742 2577 projected.go:196] Error preparing data for projected volume kube-api-access-hzwsn for pod openshift-network-diagnostics/network-check-target-9zkr2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:43.401801 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:43.401799 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn podName:4b525eeb-63a0-449b-8bb4-b89241020014 nodeName:}" failed. No retries permitted until 2026-11-02 22:47:59.401781067 +0000 UTC m=+33.508733247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzwsn" (UniqueName: "kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn") pod "network-check-target-9zkr2" (UID: "4b525eeb-63a0-449b-8bb4-b89241020014") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:47:43.534788 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.534749 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"0b29ff1fa3ed98bb0b6412645cf3cff9b4c0898344f67a566c0a744b68ed8002"} Nov 02 22:47:43.535106 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.534876 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"1e4812de674fcd21e6fa326a9ca828682e199cd2ec8d1f659766b23264796b22"} Nov 02 22:47:43.535106 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.534927 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"e7a8bbfae6e5cc7d91037f1042905f0cae7f3ec94d14d780da7e9237f1c90424"} Nov 02 22:47:43.535106 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:43.534973 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"7161d1ce1187d8be43ddf8a2f379540723047d490c4a4e0d327e4f08f24e78a9"} Nov 02 22:47:44.539951 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:44.539921 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"8ca6a9026a286e182acfc50181db398d13cbf6515122b20643695cf3e71314d1"} Nov 02 22:47:44.540326 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:44.539989 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"99b9e04ebe53894aae5acd14d61f872242d62bee762e4d1bbde9aaede19dd421"} Nov 02 22:47:45.385402 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:45.385225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:45.385564 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:45.385225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:45.385564 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:45.385484 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:45.385564 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:45.385260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:45.385658 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:45.385553 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:45.385658 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:45.385630 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:45.544824 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:45.544790 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="29c44762433dbb7892b21323ed2339dd3aba8ab5eb0d8d670be44647d12ecb2b" exitCode=0 Nov 02 22:47:45.545235 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:45.544850 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"29c44762433dbb7892b21323ed2339dd3aba8ab5eb0d8d670be44647d12ecb2b"} Nov 02 22:47:46.549455 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:46.549423 2577 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="5839343664fbeab0ae0b008a9682dfa6e241b550368abd0fbac3a3ca9a5e0ce3" exitCode=0 Nov 02 22:47:46.550085 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:46.549507 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerDied","Data":"5839343664fbeab0ae0b008a9682dfa6e241b550368abd0fbac3a3ca9a5e0ce3"} Nov 02 22:47:46.552759 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:46.552737 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"18de6194fc4941f74f2b2eac69931bb49b43d89fd53e21f3a4b19076ed5f235f"} Nov 02 22:47:47.385488 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:47.385456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:47.385655 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:47.385456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:47.385655 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:47.385570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:47.385655 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:47.385550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:47.385798 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:47.385649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:47.385798 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:47.385745 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:47.557786 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:47.557755 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-drs9r" event={"ID":"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d","Type":"ContainerStarted","Data":"7a94265c0cffb1ed0235770e0a9837e5af97bc171a5da19dddfceeba77e52d66"} Nov 02 22:47:47.680642 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:47.680557 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-drs9r" podStartSLOduration=9.048100262 podStartE2EDuration="21.680544597s" totalImagesPullingTime="12.632444335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:47.680369469 +0000 UTC m=+21.787321653" watchObservedRunningTime="2026-11-02 22:47:47.680544597 +0000 UTC m=+21.787496821" Nov 02 22:47:48.563480 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:48.563433 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" event={"ID":"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c","Type":"ContainerStarted","Data":"6ada4f23167787089da9e16f5c9e996d71ebf512b58d0c7f180b311c03c8a424"} Nov 02 22:47:48.838552 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:48.838448 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" podStartSLOduration=8.316746596 podStartE2EDuration="22.838432256s" totalImagesPullingTime="14.52168566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:48.838404182 +0000 UTC m=+22.945356370" watchObservedRunningTime="2026-11-02 22:47:48.838432256 +0000 UTC m=+22.945384430" Nov 02 22:47:49.385920 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.385888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:49.386152 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.385924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:49.386152 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.385888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:49.386152 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:49.386005 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:49.386152 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:49.386061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:49.386152 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:49.386136 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:49.566833 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.566803 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:49.567408 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.567392 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:49.567564 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.567543 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:49.583181 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.583145 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:49.583307 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:49.583273 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:47:50.617595 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.617278 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9zkr2"] Nov 02 22:47:50.618044 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.617646 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:50.618044 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:50.617727 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:50.620133 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.620099 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k9d98"] Nov 02 22:47:50.620266 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.620211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:50.620339 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:50.620288 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:50.620638 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.620607 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6zks"] Nov 02 22:47:50.620714 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:50.620690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:50.620784 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:50.620768 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:52.385116 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:52.385035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:52.385500 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:52.385035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:52.385500 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:52.385144 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:52.385500 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:52.385184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:52.385500 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:52.385265 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:52.385500 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:52.385337 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:54.094803 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:54.094757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:54.351293 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:54.351209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:54.385250 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:54.385217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:54.385396 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:54.385225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:54.385396 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:54.385319 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k9d98" podUID="1555d5cf-a68b-4168-bff4-da817535b17e" Nov 02 22:47:54.385396 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:54.385327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:54.385516 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:54.385407 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zkr2" podUID="4b525eeb-63a0-449b-8bb4-b89241020014" Nov 02 22:47:54.385516 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:54.385474 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g6zks" podUID="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" Nov 02 22:47:56.202500 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.202427 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeReady" Nov 02 22:47:56.202875 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.202600 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:47:56.315192 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.315149 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2qbd"] Nov 02 22:47:56.333783 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.333763 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gt4vq"] Nov 02 22:47:56.333915 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.333902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.344049 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.344026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.355951 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.355929 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2qbd"] Nov 02 22:47:56.359212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.359189 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gt4vq"] Nov 02 22:47:56.385702 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.385674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:56.385702 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.385700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:56.385843 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.385824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:56.389061 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.389044 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:47:56.389247 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.389231 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:47:56.389325 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.389275 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:47:56.396086 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.396069 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:47:56.396293 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.396280 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:47:56.408894 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.408872 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:47:56.408894 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.408889 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:47:56.489466 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtdm\" (UniqueName: \"kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.489466 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.489466 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.489680 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.489680 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.489680 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.489669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj27j\" (UniqueName: \"kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.493954 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.493935 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:47:56.494044 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494025 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:47:56.494112 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494092 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:47:56.494241 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494227 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:47:56.494290 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494238 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:47:56.494290 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494253 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:47:56.494351 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.494311 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:47:56.590548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.590548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.590780 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj27j\" (UniqueName: \"kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.590780 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtdm\" (UniqueName: \"kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.590780 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.590780 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.590978 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:56.590796 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:56.590978 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:56.590855 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:47:57.090833884 +0000 UTC m=+31.197786061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:47:56.590978 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.590956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.591294 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.591272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.594839 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.594721 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.651814 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.651777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtdm\" (UniqueName: \"kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:56.651990 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.651972 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj27j\" (UniqueName: \"kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:56.662883 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:56.662864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:47:57.038877 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:57.038847 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gt4vq"] Nov 02 22:47:57.042997 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:57.042968 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc738a401_2579_42f8_8876_76c38be04638.slice/crio-e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560 WatchSource:0}: Error finding container e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560: Status 404 returned error can't find the container with id e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560 Nov 02 22:47:57.094447 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:57.094419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:57.094579 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:57.094561 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:57.094628 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:57.094619 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:47:58.09460561 +0000 UTC m=+32.201557772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:47:57.584239 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:57.584203 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gt4vq" event={"ID":"c738a401-2579-42f8-8876-76c38be04638","Type":"ContainerStarted","Data":"e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560"} Nov 02 22:47:58.101644 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:58.101609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:47:58.101835 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:58.101766 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:47:58.101835 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:58.101827 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:00.101812552 +0000 UTC m=+34.208764713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:47:58.298764 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:58.298729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:58.487524 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:58.487484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:59.313089 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.313049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:59.313672 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.313145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:47:59.313672 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:59.313266 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:47:59.313672 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:47:59.313320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:31.313307063 +0000 UTC m=+65.420259225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : secret "metrics-daemon-secret" not found Nov 02 22:47:59.316341 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.316316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:59.399656 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.399627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:47:59.414315 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.414285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:59.417124 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.417104 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:59.707380 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.707300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:47:59.778360 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.771517 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k9d98"] Nov 02 22:47:59.793296 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:47:59.793271 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1555d5cf_a68b_4168_bff4_da817535b17e.slice/crio-da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a WatchSource:0}: Error finding container da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a: Status 404 returned error can't find the container with id da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a Nov 02 22:47:59.865385 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:47:59.865354 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:00.055388 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.055357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:00.079843 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.079817 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9zkr2"] Nov 02 22:48:00.082473 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:48:00.082448 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b525eeb_63a0_449b_8bb4_b89241020014.slice/crio-f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0 WatchSource:0}: Error finding container f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0: Status 404 returned error can't find the container with id f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0 Nov 02 22:48:00.119979 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.119949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:00.120150 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:00.120071 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:00.120150 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:00.120119 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:04.12010681 +0000 UTC m=+38.227058971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:00.593043 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.593011 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9zkr2" event={"ID":"4b525eeb-63a0-449b-8bb4-b89241020014","Type":"ContainerStarted","Data":"f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0"} Nov 02 22:48:00.594127 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.594102 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k9d98" event={"ID":"1555d5cf-a68b-4168-bff4-da817535b17e","Type":"ContainerStarted","Data":"da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a"} Nov 02 22:48:00.595553 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.595530 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gt4vq" event={"ID":"c738a401-2579-42f8-8876-76c38be04638","Type":"ContainerStarted","Data":"08ec7756a311150e694e3513936054f0c03ce9dce6213ec6f13ae26149b18400"} Nov 02 22:48:00.710520 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:00.710453 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gt4vq" podStartSLOduration=1.7572723639999999 podStartE2EDuration="4.710434338s" totalImagesPullingTime="2.953161974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:00.710054593 +0000 UTC m=+34.817006777" watchObservedRunningTime="2026-11-02 22:48:00.710434338 +0000 UTC m=+34.817386522" Nov 02 22:48:01.676440 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.676405 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7rxg4"] Nov 02 22:48:01.682688 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.682657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.734912 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.734882 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:48:01.784553 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.784523 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:01.785548 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.785500 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:01.786639 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.786567 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rxg4"] Nov 02 22:48:01.801337 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.801304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:01.833280 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.833242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.833280 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.833282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.833507 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.833387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwlj\" (UniqueName: \"kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.833507 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.833413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.833619 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.833505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.834853 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.834812 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:48:01.834853 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.834810 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:01.919104 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.919068 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:01.920778 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.920753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.934682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.934714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.934769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwlj\" (UniqueName: \"kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.934794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.934828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:01.934918 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:01.934980 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.434961817 +0000 UTC m=+36.541913993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.935035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935093 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.935063 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:01.935496 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:01.935123 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:01.935496 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:01.935190 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:02.43516146 +0000 UTC m=+36.542113640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : configmap "kube-rbac-proxy" not found Nov 02 22:48:01.972839 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.972803 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:48:01.973003 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:01.972902 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t4hnz\"" Nov 02 22:48:02.003210 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.003039 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.035856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.035817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.036059 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.035999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.036121 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.036057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.036208 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.036188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.073490 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.073459 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:48:02.074212 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.074163 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:02.074965 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.074946 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8"] Nov 02 22:48:02.076032 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.076007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwlj\" (UniqueName: \"kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:02.076645 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.076626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:02.077068 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.076934 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8"] Nov 02 22:48:02.136666 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.136635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.136851 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.136675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.136851 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.136708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.136851 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.136788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.137163 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.137142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.137756 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.137721 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.137974 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.137956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.159197 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.159144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.173788 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.173757 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:02.173956 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.173805 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:48:02.173956 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.173895 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cv7fc\"" Nov 02 22:48:02.198464 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.198409 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.233331 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.233296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:02.237801 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.237755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhqx\" (UniqueName: \"kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx\") pod \"migrator-66df5d496b-5p6d8\" (UID: \"16e628ec-e60c-49da-ac28-34bd947c18d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:02.308534 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.308404 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.338967 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.338934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhqx\" (UniqueName: \"kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx\") pod \"migrator-66df5d496b-5p6d8\" (UID: \"16e628ec-e60c-49da-ac28-34bd947c18d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:02.400734 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.400699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhqx\" (UniqueName: \"kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx\") pod \"migrator-66df5d496b-5p6d8\" (UID: \"16e628ec-e60c-49da-ac28-34bd947c18d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:02.440389 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.440355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:02.440563 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.440467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:02.440634 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:02.440608 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:02.440696 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:02.440670 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:03.44065185 +0000 UTC m=+37.547604016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : configmap "kube-rbac-proxy" not found Nov 02 22:48:02.441111 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:02.441089 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:02.441234 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:02.441146 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:03.441131462 +0000 UTC m=+37.548083623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:02.451045 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.450969 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.606777 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.606693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.618217 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.618188 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:02.624301 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:48:02.624268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6144b49_0219_4f29_9470_64494e6d08b8.slice/crio-991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921 WatchSource:0}: Error finding container 991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921: Status 404 returned error can't find the container with id 991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921 Nov 02 22:48:02.693723 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.693684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:02.752969 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.752918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:02.900351 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:02.900319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:03.081948 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.081859 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8"] Nov 02 22:48:03.085321 ip-10-0-135-179 kubenswrapper[2577]: W1102 22:48:03.085295 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e628ec_e60c_49da_ac28_34bd947c18d1.slice/crio-562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02 WatchSource:0}: Error finding container 562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02: Status 404 returned error can't find the container with id 562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02 Nov 02 22:48:03.449819 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.449731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:03.449995 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.449843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:03.449995 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:03.449980 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 22:48:03.450103 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:03.450042 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.450023261 +0000 UTC m=+39.556975423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : configmap "kube-rbac-proxy" not found Nov 02 22:48:03.450464 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:03.450441 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:03.450603 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:03.450502 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:05.450485836 +0000 UTC m=+39.557438010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:03.604653 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.604621 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" event={"ID":"16e628ec-e60c-49da-ac28-34bd947c18d1","Type":"ContainerStarted","Data":"562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02"} Nov 02 22:48:03.608689 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.608664 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" event={"ID":"d6144b49-0219-4f29-9470-64494e6d08b8","Type":"ContainerStarted","Data":"991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921"} Nov 02 22:48:03.655856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.655815 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5gcjg"] Nov 02 22:48:03.657825 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.657772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.710456 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.710380 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 22:48:03.710856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.710482 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 22:48:03.710856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.710523 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 22:48:03.710856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.710529 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 22:48:03.710856 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.710600 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9nscx\"" Nov 02 22:48:03.715603 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.715502 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5gcjg"] Nov 02 22:48:03.751235 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.751202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.751407 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.751299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh5z\" (UniqueName: \"kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.751407 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.751336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.852552 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.852516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh5z\" (UniqueName: \"kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.852740 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.852572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.852740 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.852617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.853738 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.853715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.856238 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.856215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.866824 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.866746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:03.913569 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.913540 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh5z\" (UniqueName: \"kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:03.973459 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:03.973377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:04.013074 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.012939 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:04.155382 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.155306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:04.155558 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:04.155441 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:04.155558 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:04.155508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:12.155492655 +0000 UTC m=+46.262444833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:04.161160 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.161127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:04.356646 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.356336 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5gcjg"] Nov 02 22:48:04.497533 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.497444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:04.614455 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.614254 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9zkr2" event={"ID":"4b525eeb-63a0-449b-8bb4-b89241020014","Type":"ContainerStarted","Data":"c8831fff99d39e41d8fc391257fe7f334ab42ae7f6525dca5e172d0db226053f"} Nov 02 22:48:04.614455 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.614362 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:48:04.615608 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.615580 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" event={"ID":"5816c8ac-9af6-4cde-bece-e51d67726b89","Type":"ContainerStarted","Data":"99d0386f3013c0f6a3ac5e14bcf1af0eaafe4d3bb827ca481e417458fb1d8a69"} Nov 02 22:48:04.730468 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:04.729602 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9zkr2" podStartSLOduration=35.263812291 podStartE2EDuration="38.729582049s" totalImagesPullingTime="3.465769758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:04.729001838 +0000 UTC m=+38.835954023" watchObservedRunningTime="2026-11-02 22:48:04.729582049 +0000 UTC m=+38.836534234" Nov 02 22:48:05.072540 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:05.072400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:05.466573 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:05.466538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:05.466744 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:05.466617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:05.467132 ip-10-0-135-179 kubenswrapper[2577]: I1102 22:48:05.467108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:05.467271 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:05.467246 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:05.467343 ip-10-0-135-179 kubenswrapper[2577]: E1102 22:48:05.467315 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:09.467295388 +0000 UTC m=+43.574247564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:05.505240 ip-10-0-135-179 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:48:05.514828 ip-10-0-135-179 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:48:05.515003 ip-10-0-135-179 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:48:05.515068 ip-10-0-135-179 systemd[1]: kubelet.service: Consumed 2.407s CPU time. Nov 02 22:48:05.538820 ip-10-0-135-179 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:48:05.601111 ip-10-0-135-179 kubenswrapper[4494]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:05.601111 ip-10-0-135-179 kubenswrapper[4494]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:48:05.601111 ip-10-0-135-179 kubenswrapper[4494]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:05.601111 ip-10-0-135-179 kubenswrapper[4494]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:48:05.603259 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603151 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.603259 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603259 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.603259 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603262 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603266 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603268 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603272 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603277 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603287 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603291 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603293 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603296 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603298 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603301 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603304 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603306 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603309 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603312 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603314 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603317 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603319 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603322 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603325 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.603356 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603328 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603331 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603333 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603336 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603338 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603341 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603345 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603350 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603353 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603355 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603359 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603362 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603364 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603367 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603370 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603373 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603376 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603381 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603385 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.603848 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603388 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603390 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603393 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603395 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603398 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603400 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603403 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603405 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603408 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603410 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603413 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603415 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603418 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603421 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603423 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603429 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603432 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603434 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603437 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.604399 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603439 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603442 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603444 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603447 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603450 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603452 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603455 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603458 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603460 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603463 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603466 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603469 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603471 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603473 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603476 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603478 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603481 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603483 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603486 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603488 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.604873 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603490 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603494 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603496 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603499 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603502 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603504 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603507 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603509 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603513 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603516 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603518 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603521 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603523 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603526 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603528 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603531 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603533 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603536 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603538 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603540 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.605387 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603543 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603547 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603550 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603553 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603555 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.603558 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604331 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604340 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604343 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604345 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604348 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604351 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604353 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604357 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604359 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604362 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604365 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604368 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604370 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.605872 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604373 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604378 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604381 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604384 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604386 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604389 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604392 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604394 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604397 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604400 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604402 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604405 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604408 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604410 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604413 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604415 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604418 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604421 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604425 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.606371 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604427 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604430 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604432 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604435 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604437 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604440 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604442 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604445 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604447 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604450 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604452 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604455 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604458 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604460 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604465 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604467 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604470 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604475 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604479 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.606837 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604482 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604485 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604488 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604490 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604493 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604495 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604498 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604500 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604503 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604505 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604508 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604510 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604513 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604516 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604518 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604521 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604523 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604526 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604528 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604532 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.607328 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604536 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604538 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604541 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604543 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604547 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604549 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604552 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604556 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604559 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604561 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604564 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604566 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604569 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604572 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604575 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604577 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604580 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604582 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604585 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.607822 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604587 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604590 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604592 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604595 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604597 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604600 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604603 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604605 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604608 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604610 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604615 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604617 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604620 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604622 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604625 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.604627 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605133 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605139 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605142 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605145 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.608298 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605150 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605152 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605155 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605157 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605160 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605162 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605165 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605185 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605189 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605194 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605198 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605202 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605205 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605208 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605210 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605213 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605216 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605218 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605221 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605224 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.608793 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605227 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605229 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605232 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605234 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605237 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605239 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605242 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605245 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605248 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605252 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605256 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605259 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605264 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605266 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605269 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605272 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605275 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605277 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605280 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.609564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605282 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605284 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605287 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605289 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605292 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605295 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605297 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605299 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605302 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605304 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605307 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605310 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605313 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605315 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605318 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605320 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605322 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605325 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605328 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605331 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.610347 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605333 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605336 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605339 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605341 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605344 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605349 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605352 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605354 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605357 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605359 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605362 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605365 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605367 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605370 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605372 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605375 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605377 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605380 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605382 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605384 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.611064 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605387 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605390 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605392 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605395 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605398 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605401 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605403 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605406 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605408 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605412 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605415 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605418 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605421 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605424 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605427 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605429 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605432 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605436 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605439 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.611589 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605441 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605444 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605446 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.605450 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605541 4494 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605547 4494 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605554 4494 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605558 4494 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605563 4494 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605566 4494 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605570 4494 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605574 4494 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605578 4494 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605581 4494 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605584 4494 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605587 4494 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605590 4494 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605593 4494 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605596 4494 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605599 4494 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605602 4494 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605605 4494 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605611 4494 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605614 4494 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:48:05.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605616 4494 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605619 4494 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605623 4494 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605626 4494 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605629 4494 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605632 4494 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605637 4494 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605640 4494 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605643 4494 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605646 4494 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605649 4494 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605652 4494 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605656 4494 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605659 4494 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605661 4494 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605664 4494 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605667 4494 flags.go:64] FLAG: --enable-server="true" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605670 4494 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605674 4494 flags.go:64] FLAG: --event-burst="100" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605677 4494 flags.go:64] FLAG: --event-qps="50" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605680 4494 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605683 4494 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605686 4494 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605690 4494 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:48:05.612765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605693 4494 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605696 4494 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605699 4494 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605702 4494 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605705 4494 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605708 4494 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605712 4494 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605715 4494 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605718 4494 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605720 4494 flags.go:64] FLAG: --feature-gates="" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605724 4494 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605728 4494 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605731 4494 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605734 4494 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605738 4494 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605742 4494 flags.go:64] FLAG: --help="false" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605745 4494 flags.go:64] FLAG: --hostname-override="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605748 4494 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605751 4494 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605753 4494 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605757 4494 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605760 4494 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605763 4494 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:48:05.613410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605765 4494 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605768 4494 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605771 4494 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605774 4494 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605777 4494 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605780 4494 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605782 4494 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605785 4494 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605788 4494 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605791 4494 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605793 4494 flags.go:64] FLAG: --lock-file="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605796 4494 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605799 4494 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605802 4494 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605807 4494 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605811 4494 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605815 4494 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605817 4494 flags.go:64] FLAG: --logging-format="text" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605820 4494 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605823 4494 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605826 4494 flags.go:64] FLAG: --manifest-url="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605829 4494 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605833 4494 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605837 4494 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605841 4494 flags.go:64] FLAG: --max-pods="110" Nov 02 22:48:05.613978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605844 4494 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605847 4494 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605850 4494 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605853 4494 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605855 4494 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605858 4494 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605870 4494 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605879 4494 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605883 4494 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605886 4494 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605888 4494 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605891 4494 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605894 4494 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605897 4494 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605899 4494 flags.go:64] FLAG: --port="10250" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605902 4494 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605905 4494 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-066f3b221f3df8e08" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605908 4494 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605911 4494 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605914 4494 flags.go:64] FLAG: --register-node="true" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605917 4494 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605921 4494 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605926 4494 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605929 4494 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605932 4494 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605936 4494 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:48:05.614587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605939 4494 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605941 4494 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605944 4494 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605947 4494 flags.go:64] FLAG: --runonce="false" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605950 4494 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605953 4494 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605956 4494 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605959 4494 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605962 4494 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605965 4494 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605968 4494 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605970 4494 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605973 4494 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605976 4494 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605979 4494 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605981 4494 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605984 4494 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605987 4494 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605990 4494 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605995 4494 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.605998 4494 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606000 4494 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606004 4494 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606007 4494 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606009 4494 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:48:05.615264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606012 4494 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606015 4494 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606018 4494 flags.go:64] FLAG: --v="2" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606026 4494 flags.go:64] FLAG: --version="false" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606030 4494 flags.go:64] FLAG: --vmodule="" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606034 4494 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606037 4494 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606187 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606191 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606194 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606197 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606200 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606203 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606205 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606208 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606210 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606213 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606215 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606218 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606221 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606225 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.615869 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606228 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606230 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606233 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606236 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606238 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606242 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606245 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606248 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606251 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606254 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606256 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606258 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606261 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606263 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606267 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606270 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606276 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606278 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606281 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.616411 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606283 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606286 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606288 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606291 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606293 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606295 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606298 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606300 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606302 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606305 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606308 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606310 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606312 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606315 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606317 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606320 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606322 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606325 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606327 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.616881 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606329 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606332 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606335 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606337 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606339 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606342 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606344 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606347 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606351 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606353 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606355 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606358 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606361 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606364 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606366 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606369 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606371 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606374 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606377 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606379 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.617409 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606382 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606384 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606387 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606389 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606392 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606394 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606397 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606399 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606402 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606404 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606407 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606409 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606411 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606415 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606418 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606420 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606423 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606425 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606428 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606430 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.617941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606434 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606437 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606439 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606441 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606444 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606447 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606450 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606452 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606455 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606457 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606459 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606462 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606464 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606467 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.606492 4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606530 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.618648 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606535 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606538 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606540 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606543 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606545 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606548 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606550 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606553 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606555 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606558 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606561 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606565 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606567 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606570 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606573 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606575 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606580 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606582 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606585 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606587 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.619057 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606590 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606593 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606595 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606598 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606601 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606603 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606605 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606608 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606610 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606613 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606615 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606617 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606620 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606622 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606624 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606627 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606629 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606632 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606634 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606637 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.619710 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606639 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606642 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606645 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606648 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606651 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606653 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606655 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606658 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606662 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606665 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606667 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606670 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606672 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606675 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606678 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606681 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606683 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606686 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606689 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606691 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.620558 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606694 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606696 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606698 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606701 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606705 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606709 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606712 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606714 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606717 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606720 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606722 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606726 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606729 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606732 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606736 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606739 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606742 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606744 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.621477 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606747 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606749 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606753 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606756 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606758 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606760 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606763 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606765 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606768 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606771 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606773 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606776 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606778 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606780 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606783 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606785 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606788 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606790 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606793 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606795 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.622263 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606797 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606800 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606802 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606806 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606808 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606811 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.606813 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.612159 4494 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.612192 4494 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.612212 4494 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.612217 4494 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612274 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612279 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612282 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612286 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612290 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612293 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612296 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612298 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612301 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.623124 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612304 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612306 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612309 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612312 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612314 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612317 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612321 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612325 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612329 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612331 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612334 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612337 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612340 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612343 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612346 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612348 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612351 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612354 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.623980 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612357 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612359 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612362 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612365 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612367 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612370 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612372 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612375 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612377 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612379 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612382 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612384 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612386 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612389 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612391 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612394 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612396 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612398 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612401 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612404 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.624811 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612407 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612409 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612412 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612415 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612417 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612420 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612423 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612425 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612428 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612430 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612433 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612435 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612438 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612440 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612443 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612445 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612447 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612450 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612452 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.625680 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612455 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612457 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612460 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612462 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612465 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612467 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612470 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612472 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612474 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612477 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612479 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612482 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612484 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612487 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612490 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612492 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612495 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612497 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612500 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612502 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.626514 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612505 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612507 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612511 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612515 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612518 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612521 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612524 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612527 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612529 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612532 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612534 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612537 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612539 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612542 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612544 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612546 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612549 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612551 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612553 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.627238 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612556 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.612584 4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612720 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612727 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612730 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612733 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612736 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612739 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612742 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612745 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612747 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612751 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612755 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612758 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612761 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.627897 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612764 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612767 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612770 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612773 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612776 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612778 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612781 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612784 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612786 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612789 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612791 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612794 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612797 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612799 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612802 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612804 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612807 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612809 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612812 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612814 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.628703 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612816 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612819 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612821 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612824 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612826 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612830 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612832 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612835 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612838 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612840 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612842 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612845 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612847 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612850 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612852 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612854 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612857 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612859 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612862 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612864 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.629687 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612866 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612869 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612871 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612874 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612877 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612879 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612882 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612884 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612887 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612889 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612892 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612895 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612897 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612899 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612902 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612907 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612909 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612911 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612914 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612917 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.630634 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612919 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612922 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612924 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612927 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612930 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612932 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612935 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612937 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612939 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612942 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612944 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612947 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612949 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612952 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612954 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612957 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612959 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612961 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612964 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612966 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.631679 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612968 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612971 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612973 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612976 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612978 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612981 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612983 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612987 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612990 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612992 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612995 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.612998 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613001 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.613024 4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613054 4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613059 4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:48:05.632694 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613063 4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613066 4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613069 4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613071 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613074 4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613077 4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613079 4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613082 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613084 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613087 4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613089 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613092 4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613094 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613097 4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613100 4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613102 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613105 4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613107 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613110 4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613113 4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:48:05.633454 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613115 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613117 4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613120 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613122 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613126 4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613128 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613131 4494 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613133 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613136 4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613139 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613142 4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613144 4494 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613147 4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613149 4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613151 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613154 4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613156 4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613159 4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613161 4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:48:05.634318 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613164 4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613167 4494 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613187 4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613190 4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613193 4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613195 4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613198 4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613200 4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613203 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613205 4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613210 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613212 4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613215 4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613217 4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613220 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613223 4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613225 4494 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613229 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613231 4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613234 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:48:05.635158 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613236 4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613239 4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613242 4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613244 4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613247 4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613249 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613252 4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613254 4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613257 4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613259 4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613262 4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613264 4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613266 4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613269 4494 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613272 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613274 4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613277 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613279 4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613282 4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613285 4494 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:48:05.636094 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613287 4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613290 4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613294 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613298 4494 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613302 4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613305 4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613308 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613311 4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613313 4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613319 4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613322 4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613324 4494 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613326 4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613330 4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613333 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613337 4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613339 4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613342 4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:48:05.636962 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613345 4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613347 4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613349 4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613352 4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613355 4494 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613358 4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:05.613360 4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.613410 4494 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614438 4494 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614491 4494 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614684 4494 server.go:1014] "Starting client certificate rotation" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614820 4494 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614883 4494 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:42:26 +0000 UTC" deadline="2028-07-02 01:50:25.806218642 +0000 UTC" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.614919 4494 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14571h2m20.191302371s" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.615254 4494 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.616403 4494 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:48:05.637509 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.617682 4494 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:48:05.637943 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.619215 4494 log.go:25] "Validated CRI v1 image API" Nov 02 22:48:05.637943 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.619912 4494 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:48:05.637943 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.622105 4494 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ccb82836-6794-421b-8035-61b033247959:/dev/nvme0n1p4 cea4f880-4b4c-474e-9a93-165bf2c7a256:/dev/nvme0n1p3] Nov 02 22:48:05.638038 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.622133 4494 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99d0386f3013c0f6a3ac5e14bcf1af0eaafe4d3bb827ca481e417458fb1d8a69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99d0386f3013c0f6a3ac5e14bcf1af0eaafe4d3bb827ca481e417458fb1d8a69/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1555d5cf-a68b-4168-bff4-da817535b17e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1555d5cf-a68b-4168-bff4-da817535b17e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e628ec-e60c-49da-ac28-34bd947c18d1/volumes/kubernetes.io~projected/kube-api-access-prhqx:{mountpoint:/var/lib/kubelet/pods/16e628ec-e60c-49da-ac28-34bd947c18d1/volumes/kubernetes.io~projected/kube-api-access-prhqx major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~projected/kube-api-access-7k2nt:{mountpoint:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~projected/kube-api-access-7k2nt major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29ac331f-070d-47b7-9c7b-3fe96663e370/volumes/kubernetes.io~projected/kube-api-access-chwlj:{mountpoint:/var/lib/kubelet/pods/29ac331f-070d-47b7-9c7b-3fe96663e370/volumes/kubernetes.io~projected/kube-api-access-chwlj major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b525eeb-63a0-449b-8bb4-b89241020014/volumes/kubernetes.io~projected/kube-api-access-hzwsn:{mountpoint:/var/lib/kubelet/pods/4b525eeb-63a0-449b-8bb4-b89241020014/volumes/kubernetes.io~projected/kube-api-access-hzwsn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555b9232-5b06-42ff-ae4f-72bf6968ef70/volumes/kubernetes.io~projected/kube-api-access-7cfjx:{mountpoint:/var/lib/kubelet/pods/555b9232-5b06-42ff-ae4f-72bf6968ef70/volumes/kubernetes.io~projected/kube-api-access-7cfjx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~projected/kube-api-access-hdh5z:{mountpoint:/var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~projected/kube-api-access-hdh5z major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a94f467-aab7-40b5-a713-356acd286516/volumes/kubernetes.io~projected/kube-api-access-pklls:{mountpoint:/var/lib/kubelet/pods/5a94f467-aab7-40b5-a713-356acd286516/volumes/kubernetes.io~projected/kube-api-access-pklls major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8/volumes/kubernetes.io~projected/kube-api-access-x4hkl:{mountpoint:/var/lib/kubelet/pods/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8/volumes/kubernetes.io~projected/kube-api-access-x4hkl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a686ab8e-a865-4f13-853b-3d43a3eef118/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a686ab8e-a865-4f13-853b-3d43a3eef118/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b586bb58-8500-4dc6-9aef-e1c58f2a39d8/volumes/kubernetes.io~projected/kube-api-access-z4phx:{mountpoint:/var/lib/kubelet/pods/b586bb58-8500-4dc6-9aef-e1c58f2a39d8/volumes/kubernetes.io~projected/kube-api-access-z4phx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c1fe43-01b8-4f60-a4da-97456b7ec024/volumes/kubernetes.io~projected/kube-api-access-xf7rg:{mountpoint:/var/lib/kubelet/pods/b7c1fe43-01b8-4f60-a4da-97456b7ec024/volumes/kubernetes.io~projected/kube-api-access-xf7rg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc97c851-78fa-479b-97cc-7e2b4061342d/volumes/kubernetes.io~projected/kube-api-access-4gtdm:{mountpoint:/var/lib/kubelet/pods/bc97c851-78fa-479b-97cc-7e2b4061342d/volumes/kubernetes.io~projected/kube-api-access-4gtdm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~projected/kube-api-access-dj27j:{mountpoint:/var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~projected/kube-api-access-dj27j major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6144b49-0219-4f29-9470-64494e6d08b8/volumes/kubernetes.io~projected/kube-api-access-dmwkn:{mountpoint:/var/lib/kubelet/pods/d6144b49-0219-4f29-9470-64494e6d08b8/volumes/kubernetes.io~projected/kube-api-access-dmwkn major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d/volumes/kubernetes.io~projected/kube-api-access-8rsjr:{mountpoint:/var/lib/kubelet/pods/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d/volumes/kubernetes.io~projected/kube-api-access-8rsjr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c23212-0707-4228-984b-9f8d412ad99c/volumes/kubernetes.io~projected/kube-api-access-lcz29:{mountpoint:/var/lib/kubelet/pods/f3c23212-0707-4228-984b-9f8d412ad99c/volumes/kubernetes.io~projected/kube-api-access-lcz29 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~projected/kube-api-access-r9tt5:{mountpoint:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~projected/kube-api-access-r9tt5 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c188b5775c55edfb8496d2be00bc7cc3f81cf06cd57d0083bacf0e8d9f39155e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/94ebb9cf190c0d58963feb4508ce6388aea362c6b530dfc4203671eb44e669d3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/58fd5f4daedbec12a59d9f79a84269d21cc3c34dc174654b71a2fb5bc1853a18/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/133342b3c1b48286a4054e370060fdc5ad95583cdd9aab8e39fa7b60a8033760/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ba6f1582dbf7f9affb892880ab29a35bd462e263c214a3d7602d0d5765c67481/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/647b6f3c56804c93920ab841be3634eae35bd0f948f417e060f1922b98de4c26/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/cd11a74c6a84f3cbc2b0a8cc85ffdc701ff726ba779708a400898867301e93c9/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/aadb57710fb8e9cd826304919626d1dd90d1f8c14ec123dedcf9acee33059ec7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/98b1913d06cc63d969dae5702ba6f75f7c2506ed80a5dc4e2d058ef61904de7d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c6f2093daec2320c0d0f350e4ccca4333a406d64348bb485d2e80931fc549a78/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/cd79ae5f054b0cde8a9fdde22ebba009b57c5e3ee07737ea08b898b5a292efbd/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/91f1a03d8388dc5b50d3181b39c8145352558d8ae1e84edd130d3636f1a1034c/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4f1b42f5ed1a64775154cbd861a638a117eb7327c0f9ab47c093bc094602af6a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4891d2cc7aebcee91a3087630ab458c3905ca27439d3265da5b53c4c86d90466/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1295e56e4a456e3f31a9e19a9d75431619ac78777cd760f1fa33195b1ee2a7f4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/01bcd6639427a4d5579519f22b2bfbd95aca604645bcf81574a006a5ad009c8b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c3080a983d77d400cf754ba75bda37e095b54ec111d27b6602080db6fdf8715e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8b17986115c74d6cf127d79039bdaf2896a8793fe1b6726849eecb58fa7e961d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/20b3988cbcab2696097a66593843c8d2946ddee30cc36c5f4bd36144fe60dc64/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9478247dc19f427f6aaa78b9a8f3f6ca73601356d6bd563337f83ab31974caa0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d4f94b1ca554ef365f81b4c9a46b60e38784e7f4429f94222c8422df6f267428/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/f1150fbf90e51a5009400d3aabd26c43a4f92257073fcb5354cd00d60b4788ed/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/8de1d8d7bf4b2cd7a7e78a917292c237844e9b990f9b2e2799ee560048922106/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/d4e6df30a5174564412aa3a36d16c54812bbf4abc05373e1c5cf157032787a09/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/d1d2a5b1c357399003da15d8c6ec4336ace7d562b17fb95e1cae4ab61c39beee/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/c53f7115adfae742fc49bfba3d62088da0eeb2917da3a12d0792f25a3c5306b3/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/f88a161b739e72080d4195cff42fa1fabbf088fe5429a5fb39c8b36e132cb8f7/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/64b0afee694eaf6272004cbf3c420f2e69d25c24370d03d25762e8b21d395ed7/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/ec1d4a764259a612916e508935f7c16eb521a8b65c60033ca723e2b139fd3f8b/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/c615e9073ce4269386ac9f312bd693ec45b805a1131b056849a350eb4c315665/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/ab387e6f5da340a9f4cd4751073ae75492d3ea48c0b5ed6fea29c7fa8dde5fd3/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/fcefa042becb5153654da51710241d600a9c36516a0ef41625144b38142840e5/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/8ec179c0c1444ac6c38eb2e66137b75abdc1eadd1851e31f8ade85997f67bb39/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/27f0769a80a51a915bd056437f1e52bd72c09b1990d316723aebff049d63e76b/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/6028311a6bcd453c2754311774cc21cef4fc56285a6c3818a8fafa352e3450b6/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/73eff04ead8c7caee3d935f4ceb6b90c8b96e598403cd096d09b17fdf6ce0345/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0cf62c783ff16d3c43f7b3b022e4f002a3cdb4fc84e39b1dda8193971fd30464/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d28cd19f312cf3273dd6d963346180570e22d5ce0bfcd81c39e5500693144055/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ca1ee74794898a4d4f35a606d8d5960af7b5473394bde1f1c0891cf540ff546f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0cd993d1ee7766c1517fea688ae567de31ca2ca97047afabfedf004fcc137970/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3267d3baf14ec51996b9f40cc32080426434cac47e6b545edd34fb3aca96e471/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5e24b83830593234c09eb2d9dcbea89248be9bb0e19cda6a21f66bb8c96572c2/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.633857 4494 manager.go:217] Machine: {Timestamp:2026-11-02 22:48:05.633047446 +0000 UTC m=+0.062917514 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3160628 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec225e6d36ec6733b43309850ba73356 SystemUUID:ec225e6d-36ec-6733-b433-09850ba73356 BootID:84bb8911-e988-4819-84e3-84ae636cac68 Filesystems:[{Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d/volumes/kubernetes.io~projected/kube-api-access-8rsjr DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b586bb58-8500-4dc6-9aef-e1c58f2a39d8/volumes/kubernetes.io~projected/kube-api-access-z4phx DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/227e156ad9ab8afef3c1c1acffaa5db07b0bef7b6dc95dee44fc9fdbb23f4a9d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9459115bed22a6ca0f2ed8cd5ec89f18674dea2b59dc1a142306da2dd81736/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/555b9232-5b06-42ff-ae4f-72bf6968ef70/volumes/kubernetes.io~projected/kube-api-access-7cfjx DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da640bddb05f1a5e4f80c457d7cbc9388be93c317169dd56d9845ae98b50cd4a/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b525eeb-63a0-449b-8bb4-b89241020014/volumes/kubernetes.io~projected/kube-api-access-hzwsn DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c738a401-2579-42f8-8876-76c38be04638/volumes/kubernetes.io~projected/kube-api-access-dj27j DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1555d5cf-a68b-4168-bff4-da817535b17e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99d0386f3013c0f6a3ac5e14bcf1af0eaafe4d3bb827ca481e417458fb1d8a69/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c1fe43-01b8-4f60-a4da-97456b7ec024/volumes/kubernetes.io~projected/kube-api-access-xf7rg DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29ac331f-070d-47b7-9c7b-3fe96663e370/volumes/kubernetes.io~projected/kube-api-access-chwlj DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5816c8ac-9af6-4cde-bece-e51d67726b89/volumes/kubernetes.io~projected/kube-api-access-hdh5z DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6144b49-0219-4f29-9470-64494e6d08b8/volumes/kubernetes.io~projected/kube-api-access-dmwkn DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volumes/kubernetes.io~projected/kube-api-access-7k2nt DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d86f37af3bbfa3e7b19b66d0664bdf45296efc0b1420653fd08caf8c90838acf/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a94f467-aab7-40b5-a713-356acd286516/volumes/kubernetes.io~projected/kube-api-access-pklls DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8/volumes/kubernetes.io~projected/kube-api-access-x4hkl DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc97c851-78fa-479b-97cc-7e2b4061342d/volumes/kubernetes.io~projected/kube-api-access-4gtdm DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e30f8ef0187ae3d32be8cc1212ba6cc247edcba7cfad241f3ef78b0cb58d4560/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9879a7e83638ac6941fc6dcfbcb152958dec5d7d92b5a65bcfd816e3fc09844e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/16e628ec-e60c-49da-ac28-34bd947c18d1/volumes/kubernetes.io~projected/kube-api-access-prhqx DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562c1862f0c61cb965c8b3394595ed8adc488f9235217cfbeccd60488ed14e02/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca2f3a1813b397f0a3a90db2716d438de9ebe6acdba7dfd83081a10ea4a1cc62/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f16b0e01edafa0553d604543885cc63cb4d4dfea585b2dd3dc026ffcbbb478c0/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5d4bb81c3d2446144f0c5bd86111cc1411a3daae67e30a1249ee93e30c8b1d/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a92053503b5916883d1c9b01b33acfd9b50eed2d84b27f2f0cc453dc1b49f9b1/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a686ab8e-a865-4f13-853b-3d43a3eef118/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c23212-0707-4228-984b-9f8d412ad99c/volumes/kubernetes.io~projected/kube-api-access-lcz29 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f726490e-b32d-43ec-a286-750ec26f808d/volumes/kubernetes.io~projected/kube-api-access-r9tt5 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf28ca367333f546420c40403681dd0a27b442914f331e492a9cf789cbe8c90b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:562c1862f0c61cb MacAddress:46:8c:6c:86:94:a1 Speed:10000 Mtu:8901} {Name:991ef4f3dd84027 MacAddress:96:a7:a6:c5:17:58 Speed:10000 Mtu:8901} {Name:99d0386f3013c0f MacAddress:f6:23:29:eb:24:27 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:47:20:e7:6a:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:96:af:1d:6f:83:1e Speed:0 Mtu:8901} {Name:da640bddb05f1a5 MacAddress:1a:20:a9:d9:d8:dd Speed:10000 Mtu:8901} {Name:e30f8ef0187ae3d MacAddress:72:64:c9:69:61:6b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:47:20:e7:6a:57 Speed:0 Mtu:9001} {Name:f16b0e01edafa05 MacAddress:26:9f:97:5c:2e:a1 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f2:5a:90:9d:86:49 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:f6:c4:97:77:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634356 4494 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634419 4494 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634747 4494 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634773 4494 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-179.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634964 4494 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634975 4494 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.634990 4494 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635016 4494 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635117 4494 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635347 4494 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635486 4494 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635531 4494 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635546 4494 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635561 4494 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635575 4494 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.635587 4494 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.636373 4494 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.636385 4494 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:05.638197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.636900 4494 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:48:05.645365 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645351 4494 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:48:05.645365 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645365 4494 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:48:05.645479 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645463 4494 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:48:05.645675 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645663 4494 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:48:05.645717 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645693 4494 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:48:05.645801 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645792 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645811 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645821 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645827 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645832 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645838 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:48:05.645846 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645845 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645850 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645858 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645872 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645883 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645896 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645924 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:48:05.646068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.645929 4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:48:05.649382 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.649368 4494 server.go:1263] "Started kubelet" Nov 02 22:48:05.649465 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.649444 4494 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:48:05.649549 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.649472 4494 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:48:05.649601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.649573 4494 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:48:05.652208 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.649852 4494 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:48:05.649960 ip-10-0-135-179 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:48:05.653821 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.653797 4494 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:48:05.653957 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.653939 4494 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:48:05.654290 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654250 4494 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:48:05.654453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654383 4494 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:42:27 +0000 UTC" deadline="2028-04-09 03:34:17.352485458 +0000 UTC" Nov 02 22:48:05.654453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654429 4494 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12556h46m11.698059874s" Nov 02 22:48:05.654633 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654467 4494 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:48:05.654633 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654473 4494 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:48:05.654633 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.654573 4494 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:48:05.655915 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.655374 4494 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" Nov 02 22:48:05.657150 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657133 4494 factory.go:153] Registering CRI-O factory Nov 02 22:48:05.657249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657154 4494 factory.go:223] Registration of the crio container factory successfully Nov 02 22:48:05.657249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657163 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" volumeName="kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr" seLinuxMountContext="" Nov 02 22:48:05.657249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657229 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e628ec-e60c-49da-ac28-34bd947c18d1" volumeName="kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx" seLinuxMountContext="" Nov 02 22:48:05.657249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657245 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb3950-1c7d-4855-a7b7-eaf1b6fa095c" volumeName="kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657262 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a94f467-aab7-40b5-a713-356acd286516" volumeName="kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657276 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc97c851-78fa-479b-97cc-7e2b4061342d" volumeName="kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657293 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29ac331f-070d-47b7-9c7b-3fe96663e370" volumeName="kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657306 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555b9232-5b06-42ff-ae4f-72bf6968ef70" volumeName="kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657319 4494 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657332 4494 factory.go:55] Registering systemd factory Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657341 4494 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657319 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a686ab8e-a865-4f13-853b-3d43a3eef118" volumeName="kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657387 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6144b49-0219-4f29-9470-64494e6d08b8" volumeName="kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657414 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c23212-0707-4228-984b-9f8d412ad99c" volumeName="kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657430 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f726490e-b32d-43ec-a286-750ec26f808d" volumeName="kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657444 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f726490e-b32d-43ec-a286-750ec26f808d" volumeName="kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5" seLinuxMountContext="" Nov 02 22:48:05.657456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657360 4494 factory.go:103] Registering Raw factory Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657486 4494 manager.go:1199] Started watching for new ooms in manager Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657458 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5816c8ac-9af6-4cde-bece-e51d67726b89" volumeName="kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657504 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1555d5cf-a68b-4168-bff4-da817535b17e" volumeName="kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657521 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29ac331f-070d-47b7-9c7b-3fe96663e370" volumeName="kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657534 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a94f467-aab7-40b5-a713-356acd286516" volumeName="kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657548 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b586bb58-8500-4dc6-9aef-e1c58f2a39d8" volumeName="kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657560 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6144b49-0219-4f29-9470-64494e6d08b8" volumeName="kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657574 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29ac331f-070d-47b7-9c7b-3fe96663e370" volumeName="kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657588 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b525eeb-63a0-449b-8bb4-b89241020014" volumeName="kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657601 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5816c8ac-9af6-4cde-bece-e51d67726b89" volumeName="kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657613 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="938e7ec8-af54-4d4f-b0c7-507a87ebb9e8" volumeName="kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657626 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c1fe43-01b8-4f60-a4da-97456b7ec024" volumeName="kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657640 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c738a401-2579-42f8-8876-76c38be04638" volumeName="kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657653 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" volumeName="kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657667 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c23212-0707-4228-984b-9f8d412ad99c" volumeName="kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657680 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb3950-1c7d-4855-a7b7-eaf1b6fa095c" volumeName="kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657693 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb3950-1c7d-4855-a7b7-eaf1b6fa095c" volumeName="kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657711 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555b9232-5b06-42ff-ae4f-72bf6968ef70" volumeName="kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657725 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6144b49-0219-4f29-9470-64494e6d08b8" volumeName="kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657739 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" volumeName="kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657751 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" volumeName="kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657764 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c1fe43-01b8-4f60-a4da-97456b7ec024" volumeName="kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657778 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c23212-0707-4228-984b-9f8d412ad99c" volumeName="kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657792 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb3950-1c7d-4855-a7b7-eaf1b6fa095c" volumeName="kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657804 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5816c8ac-9af6-4cde-bece-e51d67726b89" volumeName="kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657816 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c738a401-2579-42f8-8876-76c38be04638" volumeName="kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657824 4494 manager.go:319] Starting recovery of all containers Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657830 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6144b49-0219-4f29-9470-64494e6d08b8" volumeName="kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn" seLinuxMountContext="" Nov 02 22:48:05.657847 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657843 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f726490e-b32d-43ec-a286-750ec26f808d" volumeName="kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp" seLinuxMountContext="" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657858 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cb3950-1c7d-4855-a7b7-eaf1b6fa095c" volumeName="kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides" seLinuxMountContext="" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657885 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a686ab8e-a865-4f13-853b-3d43a3eef118" volumeName="kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs" seLinuxMountContext="" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657897 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc97c851-78fa-479b-97cc-7e2b4061342d" volumeName="kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir" seLinuxMountContext="" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657910 4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc97c851-78fa-479b-97cc-7e2b4061342d" volumeName="kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm" seLinuxMountContext="" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657923 4494 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:48:05.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.657930 4494 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:48:05.669481 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.669447 4494 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:48:05.685563 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.685528 4494 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:48:05.685563 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.685560 4494 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:48:05.685737 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.685577 4494 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:48:05.685737 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.685615 4494 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:48:05.708282 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708251 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="5839343664fbeab0ae0b008a9682dfa6e241b550368abd0fbac3a3ca9a5e0ce3" exitCode=0 Nov 02 22:48:05.708282 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708276 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="29c44762433dbb7892b21323ed2339dd3aba8ab5eb0d8d670be44647d12ecb2b" exitCode=0 Nov 02 22:48:05.708282 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708283 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="c86245df72270d63d9f00827da287b8570f4d3668d9281d13c818381c321a44a" exitCode=0 Nov 02 22:48:05.708282 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708290 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="aeeca20ae6fc636a8f0d2917fe60d23c2023a227a43350c1f2e3c73a4adb97a5" exitCode=0 Nov 02 22:48:05.708577 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708296 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="57e5981938fb44fd27151fe3758273b8b23321b6774a2cf1c67b0e668e7f639d" exitCode=0 Nov 02 22:48:05.708577 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.708302 4494 generic.go:356] "Generic (PLEG): container finished" podID="e76d6a1d-5b9b-4bbe-8848-f1400d29d16d" containerID="3c673435694cd266834fa435aed9c85afcea1a30ceed75732b3a63c88f1cc820" exitCode=0 Nov 02 22:48:05.716226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.716187 4494 generic.go:356] "Generic (PLEG): container finished" podID="3bd1027324a8515ba7169fd6975c2633" containerID="c20d8d51aab963c612d198855425998b9ec2b8e5626160e9e31471573e163b69" exitCode=0 Nov 02 22:48:05.727892 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.727870 4494 manager.go:324] Recovery completed Nov 02 22:48:05.740432 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.740412 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.741544 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.741528 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.741651 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.741559 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.741651 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.741569 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.742929 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.742908 4494 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:48:05.742929 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.742928 4494 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:48:05.743022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.742948 4494 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:48:05.743022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.742920 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.743107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743089 4494 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:48:05.743139 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743110 4494 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:48:05.743139 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743134 4494 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:48:05.743246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743142 4494 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:48:05.743246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743163 4494 policy_none.go:50] "Start" Nov 02 22:48:05.743246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743190 4494 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:48:05.743246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743201 4494 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:48:05.743397 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743343 4494 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:48:05.743397 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743353 4494 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:48:05.743397 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743360 4494 policy_none.go:44] "Start" Nov 02 22:48:05.743835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743822 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.743894 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743849 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.743894 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.743860 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.748874 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.748845 4494 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:48:05.751643 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.751629 4494 manager.go:341] "Starting Device Plugin manager" Nov 02 22:48:05.751703 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.751654 4494 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:48:05.751703 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.751663 4494 server.go:87] "Starting device plugin registration server" Nov 02 22:48:05.751975 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.751963 4494 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:48:05.752023 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.751976 4494 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:48:05.752134 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.752121 4494 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:48:05.752271 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.752238 4494 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:48:05.752326 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.752279 4494 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:48:05.752326 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.752286 4494 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:48:05.752428 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.752326 4494 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:48:05.752269732Z","UUID":"a0554f14-ce9c-40fe-8886-907efd8351e4","Handler":null,"Name":"","Endpoint":""} Nov 02 22:48:05.753720 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.753701 4494 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:48:05.753820 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.753738 4494 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:48:05.754328 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.754303 4494 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:48:05.754401 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.754351 4494 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" Nov 02 22:48:05.786346 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.786285 4494 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal"] Nov 02 22:48:05.786450 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.786418 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.787347 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.787323 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.787420 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.787351 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.787420 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.787362 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.787599 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.787589 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.787741 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.787731 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.788353 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788338 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.788434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788361 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.788434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788371 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.788434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788379 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.788434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788403 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.788434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788414 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.788565 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.788494 4494 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.788696 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788662 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"b8bcc47f51109dce7950aafef82fe134","Type":"ContainerStarted","Data":"c4fc8d03a4d56851e998a2b151b68aaa4f2e7266e65055ce9f0fc90486bba111"} Nov 02 22:48:05.788726 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788709 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"b8bcc47f51109dce7950aafef82fe134","Type":"ContainerStarted","Data":"00659681a8b7f6626ee0437be12f68b69f22e7bf169a48f35c26a8cd1002e006"} Nov 02 22:48:05.788753 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788728 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerStarted","Data":"47b42536e078b1a4dec9659c0d8a24fdac8d1924e7bd401d97ae4a962c31e3d4"} Nov 02 22:48:05.788753 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788749 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.788834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788749 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerDied","Data":"c20d8d51aab963c612d198855425998b9ec2b8e5626160e9e31471573e163b69"} Nov 02 22:48:05.788866 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.788857 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" event={"ID":"3bd1027324a8515ba7169fd6975c2633","Type":"ContainerStarted","Data":"7a72fa2ae6bfbd0c2b5c0db8b9b1dfd2a3df6fe41ff8512bb13ea341948812c7"} Nov 02 22:48:05.789549 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.789535 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.789593 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.789566 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.789593 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.789582 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.789713 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.789700 4494 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.us-west-1.compute.internal\" not found" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.849948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.849917 4494 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:05.849948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.849922 4494 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:48:05.853006 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.852983 4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:48:05.854074 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.854050 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:48:05.854161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.854084 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:48:05.854161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.854094 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 22:48:05.854161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.854126 4494 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.856239 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.856202 4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.900734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.900709 4494 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:05.900734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.900712 4494 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:05.900881 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.900777 4494 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:48:05.941664 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.941635 4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:05.941814 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:05.941677 4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.941814 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.941692 4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.950877 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.950855 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.950931 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.950882 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.950931 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.950901 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.994321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.994295 4494 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.994436 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.994358 4494 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:05.994929 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.994915 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeNotReady" Nov 02 22:48:05.994986 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:05.994957 4494 setters.go:546] "Node became not ready" node="ip-10-0-135-179.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T22:48:05Z","lastTransitionTime":"2026-11-02T22:48:05Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 22:48:06.015013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.014986 4494 nodeinfomanager.go:471] existing CSINode "ip-10-0-135-179.us-west-1.compute.internal" is owned by different node (oldNodeID="c82795fd-f574-4ca7-9ff1-b6ab66f080ce", newNodeID=""), cleaning up... Nov 02 22:48:06.019183 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.019160 4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:06.019231 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.019213 4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051617 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051745 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051644 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051745 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051665 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051745 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051709 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bcc47f51109dce7950aafef82fe134-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"b8bcc47f51109dce7950aafef82fe134\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051849 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051744 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.051849 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.051749 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bd1027324a8515ba7169fd6975c2633-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" (UID: \"3bd1027324a8515ba7169fd6975c2633\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.636717 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.636682 4494 apiserver.go:52] "Watching apiserver" Nov 02 22:48:06.727470 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.727444 4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.727622 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.727579 4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.742159 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.742129 4494 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:48:06.742713 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.742683 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-rrqq2","kube-system/konnectivity-agent-792jn","openshift-dns/dns-default-t2qbd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-5gcjg","openshift-ingress-canary/ingress-canary-gt4vq","openshift-insights/insights-runtime-extractor-7rxg4","openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8","openshift-multus/multus-cb9nm","openshift-insights/periodic-gathering-rjlhh-4swj9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw","openshift-network-diagnostics/network-check-target-9zkr2","kube-system/global-pull-secret-syncer-k9d98","openshift-dns/node-resolver-7jpbd","openshift-multus/multus-additional-cni-plugins-drs9r","kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-t92fq","openshift-cluster-node-tuning-operator/tuned-8fszs","openshift-multus/network-metrics-daemon-g6zks","openshift-image-registry/node-ca-px8c4"] Nov 02 22:48:06.743667 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.743647 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:06.746899 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.746881 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.748650 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.748632 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.756354 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.756339 4494 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:48:06.788366 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.788344 4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:06.788466 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.788394 4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.788530 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.788454 4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:48:06.788644 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.788518 4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.us-west-1.compute.internal" Nov 02 22:48:06.797247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.797219 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:48:06.797521 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.797403 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2dkz\"" Nov 02 22:48:06.797521 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.797430 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d58gp\"" Nov 02 22:48:06.797622 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.797525 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8wwr\"" Nov 02 22:48:06.797622 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.797541 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4f84\"" Nov 02 22:48:06.843977 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.843949 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:48:06.843977 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.843965 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844137 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.843984 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844137 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844004 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:48:06.844137 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844066 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844216 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:48:06.844339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844228 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.844339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844316 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844323 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.844458 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844336 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:48:06.844458 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844348 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.844594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844578 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844578 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844592 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844840 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.844898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844851 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844855 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.844898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844884 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.844898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844885 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844889 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844914 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844935 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844957 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.844960 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845027 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845072 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845192 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845235 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845243 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845255 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845275 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:48:06.845312 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845276 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:48:06.845731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845327 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845335 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:48:06.845731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845389 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:48:06.845731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845492 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:48:06.845731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.845534 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:48:06.847904 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.847890 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:48:06.855941 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.855920 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.855956 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.855986 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856009 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856064 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856121 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsjr\" (UniqueName: \"kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856211 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-script-lib\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856215 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.856414 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856327 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856414 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856356 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856419 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklls\" (UniqueName: \"kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.856490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856442 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856450 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.856490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856475 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.856617 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856499 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.856617 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856554 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.856617 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856599 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-binary-copy\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.856712 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856697 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856753 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856737 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.856782 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856763 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.856810 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856779 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.856810 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856804 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tt5\" (UniqueName: \"kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.856871 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856836 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:48:06.856906 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856896 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:48:06.857011 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.856992 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.857071 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857055 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.857103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857083 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.857158 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857100 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.857158 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857114 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.857276 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857160 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.857276 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857165 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc97c851-78fa-479b-97cc-7e2b4061342d-config-volume\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.857276 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857223 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.857276 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857265 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857270 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc97c851-78fa-479b-97cc-7e2b4061342d-tmp-dir\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857306 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2nt\" (UniqueName: \"kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857345 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857361 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.857391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857384 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.857594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857467 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovnkube-config\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.857594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857481 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.857594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857507 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.857594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857532 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:48:06.857594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857567 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857670 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857712 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857736 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857767 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:48:06.857834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857803 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a686ab8e-a865-4f13-853b-3d43a3eef118-konnectivity-ca\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:48:06.858025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857882 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4phx\" (UniqueName: \"kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:06.858025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857929 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtdm\" (UniqueName: \"kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.858025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.857986 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858109 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858063 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.858109 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858080 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858109 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858095 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.858210 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858109 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858210 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858145 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858220 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.858264 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858238 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.858324 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858254 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7rg\" (UniqueName: \"kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.858324 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858278 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858324 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858301 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858340 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-tmp\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.858411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858343 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858383 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858403 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858420 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.858517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858459 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858486 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29ac331f-070d-47b7-9c7b-3fe96663e370-data-volume\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.858597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858516 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.858597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858552 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.858597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858578 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555b9232-5b06-42ff-ae4f-72bf6968ef70-tmp-dir\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.858597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858588 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.858708 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858613 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.858708 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858676 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858713 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.858785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858719 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.858785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858735 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.858785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858752 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:06.858919 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858790 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.858919 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858795 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.858919 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858819 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.858919 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858826 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.858919 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858857 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.859087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858954 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-env-overrides\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.859087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858961 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-cni-binary-copy\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.859087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858967 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.859087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.858997 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.859087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859039 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859089 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj27j\" (UniqueName: \"kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859120 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859148 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859194 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh5z\" (UniqueName: \"kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859222 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859251 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz29\" (UniqueName: \"kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.859329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859251 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c23212-0707-4228-984b-9f8d412ad99c-multus-daemon-config\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-cabundle\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859359 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859393 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859409 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhqx\" (UniqueName: \"kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx\") pod \"migrator-66df5d496b-5p6d8\" (UID: \"16e628ec-e60c-49da-ac28-34bd947c18d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859425 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859448 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hkl\" (UniqueName: \"kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859471 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859491 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f726490e-b32d-43ec-a286-750ec26f808d-etc-tuned\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859496 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.859583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859531 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwlj\" (UniqueName: \"kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859638 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfjx\" (UniqueName: \"kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859670 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859695 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859737 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859862 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859895 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859919 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859946 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a94f467-aab7-40b5-a713-356acd286516-serviceca\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859964 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.860013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.859992 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.860439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.860090 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.860439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.860138 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.860439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.860203 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.860439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.860232 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7c1fe43-01b8-4f60-a4da-97456b7ec024-iptables-alerter-script\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.895829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.895770 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 22:48:06.897102 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897082 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:48:06.897211 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897092 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:48:06.897211 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897149 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 22:48:06.897211 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897161 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:48:06.897211 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897200 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897291 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897295 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wk6k8\"" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897293 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897356 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c738a401-2579-42f8-8876-76c38be04638-cert\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897395 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bxrdd\"" Nov 02 22:48:06.897409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897404 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf97t\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897446 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-77tzt\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897454 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9nscx\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897459 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jqgm\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897453 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t4hnz\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897476 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cv7fc\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897488 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pl59x\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897498 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zfnv8\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897507 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnmtx\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897508 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb4zk\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897514 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5z48j\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897523 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:48:06.897627 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.897564 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cz5vj\"" Nov 02 22:48:06.898907 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.898891 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.899916 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.899901 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-ovn-node-metrics-cert\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.899974 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.899947 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5816c8ac-9af6-4cde-bece-e51d67726b89-signing-key\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.901101 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.901085 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1555d5cf-a68b-4168-bff4-da817535b17e-original-pull-secret\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.907292 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.907273 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a686ab8e-a865-4f13-853b-3d43a3eef118-agent-certs\") pod \"konnectivity-agent-792jn\" (UID: \"a686ab8e-a865-4f13-853b-3d43a3eef118\") " pod="kube-system/konnectivity-agent-792jn" Nov 02 22:48:06.916026 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.915997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklls\" (UniqueName: \"kubernetes.io/projected/5a94f467-aab7-40b5-a713-356acd286516-kube-api-access-pklls\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.916270 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.916255 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2nt\" (UniqueName: \"kubernetes.io/projected/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-kube-api-access-7k2nt\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.916311 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.916257 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tt5\" (UniqueName: \"kubernetes.io/projected/f726490e-b32d-43ec-a286-750ec26f808d-kube-api-access-r9tt5\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.916559 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.916541 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsjr\" (UniqueName: \"kubernetes.io/projected/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-kube-api-access-8rsjr\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.916976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.916957 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwsn\" (UniqueName: \"kubernetes.io/projected/4b525eeb-63a0-449b-8bb4-b89241020014-kube-api-access-hzwsn\") pod \"network-check-target-9zkr2\" (UID: \"4b525eeb-63a0-449b-8bb4-b89241020014\") " pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:48:06.918289 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.918275 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtdm\" (UniqueName: \"kubernetes.io/projected/bc97c851-78fa-479b-97cc-7e2b4061342d-kube-api-access-4gtdm\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.918399 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.918370 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7rg\" (UniqueName: \"kubernetes.io/projected/b7c1fe43-01b8-4f60-a4da-97456b7ec024-kube-api-access-xf7rg\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.918437 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.918415 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4phx\" (UniqueName: \"kubernetes.io/projected/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-kube-api-access-z4phx\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:06.919306 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.919287 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh5z\" (UniqueName: \"kubernetes.io/projected/5816c8ac-9af6-4cde-bece-e51d67726b89-kube-api-access-hdh5z\") pod \"service-ca-5fffc97bdf-5gcjg\" (UID: \"5816c8ac-9af6-4cde-bece-e51d67726b89\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" Nov 02 22:48:06.919884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.919865 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz29\" (UniqueName: \"kubernetes.io/projected/f3c23212-0707-4228-984b-9f8d412ad99c-kube-api-access-lcz29\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.920003 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.919987 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hkl\" (UniqueName: \"kubernetes.io/projected/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kube-api-access-x4hkl\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.920046 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.919988 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj27j\" (UniqueName: \"kubernetes.io/projected/c738a401-2579-42f8-8876-76c38be04638-kube-api-access-dj27j\") pod \"ingress-canary-gt4vq\" (UID: \"c738a401-2579-42f8-8876-76c38be04638\") " pod="openshift-ingress-canary/ingress-canary-gt4vq" Nov 02 22:48:06.921155 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.921135 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfjx\" (UniqueName: \"kubernetes.io/projected/555b9232-5b06-42ff-ae4f-72bf6968ef70-kube-api-access-7cfjx\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.921339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.921323 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwlj\" (UniqueName: \"kubernetes.io/projected/29ac331f-070d-47b7-9c7b-3fe96663e370-kube-api-access-chwlj\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.925209 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.925159 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhqx\" (UniqueName: \"kubernetes.io/projected/16e628ec-e60c-49da-ac28-34bd947c18d1-kube-api-access-prhqx\") pod \"migrator-66df5d496b-5p6d8\" (UID: \"16e628ec-e60c-49da-ac28-34bd947c18d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p6d8" Nov 02 22:48:06.925348 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.925331 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"periodic-gathering-rjlhh-4swj9\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:06.961109 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961086 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.961215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961113 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.961215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961128 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961145 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961161 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.961215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961197 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961221 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961228 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-cnibin\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961253 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-node-log\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961228 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-ovn\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961291 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961308 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-system-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961322 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-conf\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961327 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.961346 4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961360 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-socket-dir-parent\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961365 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961370 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-etc-kubernetes\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961392 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961345 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.961432 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:07.461403692 +0000 UTC m=+1.891273547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961446 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-systemd-units\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961452 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961476 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961478 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-kubernetes\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961490 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961507 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961529 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961553 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961562 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-run\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961573 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-log-socket\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a94f467-aab7-40b5-a713-356acd286516-host\") pod \"node-ca-px8c4\" (UID: \"5a94f467-aab7-40b5-a713-356acd286516\") " pod="openshift-image-registry/node-ca-px8c4" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961576 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961611 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29ac331f-070d-47b7-9c7b-3fe96663e370-crio-socket\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961620 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysctl-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961626 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-kubelet\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961621 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961658 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961675 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-host\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961660 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961693 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961712 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-device-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961733 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961757 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961732 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-conf-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961786 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961801 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961815 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961830 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961845 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961860 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961893 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-sys\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961904 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-slash\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961909 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961919 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-system-cni-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961931 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961940 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555b9232-5b06-42ff-ae4f-72bf6968ef70-hosts-file\") pod \"node-resolver-7jpbd\" (UID: \"555b9232-5b06-42ff-ae4f-72bf6968ef70\") " pod="openshift-dns/node-resolver-7jpbd" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961943 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-multus-certs\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961961 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961982 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.961984 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-netd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962001 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-var-lib-kubelet\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962002 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-modprobe-d\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962019 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962028 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-lib-modules\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962054 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962062 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-sys-fs\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962102 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-run-systemd\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962105 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-multus-cni-dir\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.962107 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962137 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962163 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962194 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962211 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-netns\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962215 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962253 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-hostroot\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962239 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-netns\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962259 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-systemd\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962269 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962294 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962318 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962329 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-var-lib-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962345 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962354 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-cni-bin\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962368 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-etc-openvswitch\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962376 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962383 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-run-k8s-cni-cncf-io\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962400 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962422 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962447 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962458 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-bin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962475 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962487 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-kubelet-config\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962501 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.962528 4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962531 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962539 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-socket-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.962534 4494 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962554 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.962570 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:07.462557889 +0000 UTC m=+1.892427728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962575 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-host-var-lib-cni-multus\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:06.962590 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:07.46257935 +0000 UTC m=+1.892449189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : secret "metrics-daemon-secret" not found Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962590 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-registration-dir\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962605 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f726490e-b32d-43ec-a286-750ec26f808d-etc-sysconfig\") pod \"tuned-8fszs\" (UID: \"f726490e-b32d-43ec-a286-750ec26f808d\") " pod="openshift-cluster-node-tuning-operator/tuned-8fszs" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962606 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962638 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/938e7ec8-af54-4d4f-b0c7-507a87ebb9e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptxmw\" (UID: \"938e7ec8-af54-4d4f-b0c7-507a87ebb9e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptxmw" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962646 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962673 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962699 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-kubelet\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962700 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962728 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962736 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7c1fe43-01b8-4f60-a4da-97456b7ec024-host-slash\") pod \"iptables-alerter-rrqq2\" (UID: \"b7c1fe43-01b8-4f60-a4da-97456b7ec024\") " pod="openshift-network-operator/iptables-alerter-rrqq2" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962745 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962748 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e76d6a1d-5b9b-4bbe-8848-f1400d29d16d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-drs9r\" (UID: \"e76d6a1d-5b9b-4bbe-8848-f1400d29d16d\") " pod="openshift-multus/multus-additional-cni-plugins-drs9r" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962758 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962777 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c23212-0707-4228-984b-9f8d412ad99c-cnibin\") pod \"multus-cb9nm\" (UID: \"f3c23212-0707-4228-984b-9f8d412ad99c\") " pod="openshift-multus/multus-cb9nm" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962834 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1555d5cf-a68b-4168-bff4-da817535b17e-dbus\") pod \"global-pull-secret-syncer-k9d98\" (UID: \"1555d5cf-a68b-4168-bff4-da817535b17e\") " pod="kube-system/global-pull-secret-syncer-k9d98" Nov 02 22:48:06.963607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.962778 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20cb3950-1c7d-4855-a7b7-eaf1b6fa095c-host-run-ovn-kubernetes\") pod \"ovnkube-node-t92fq\" (UID: \"20cb3950-1c7d-4855-a7b7-eaf1b6fa095c\") " pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:06.974010 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:06.973989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:07.050470 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:07.050451 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:48:07.467347 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:07.467312 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:07.467387 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:07.467412 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467451 4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467513 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:08.467497443 +0000 UTC m=+2.897367283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467514 4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:07.467548 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467522 4494 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:48:07.467752 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467562 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:08.467546621 +0000 UTC m=+2.897416461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:07.467752 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:07.467583 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:08.467568412 +0000 UTC m=+2.897438252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : secret "metrics-daemon-secret" not found Nov 02 22:48:08.476187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:08.476140 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:08.476255 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:08.476288 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476426 4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476480 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:10.476465527 +0000 UTC m=+4.906335382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476528 4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476551 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:10.476544442 +0000 UTC m=+4.906414281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476594 4494 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:48:08.476646 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:08.476617 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:10.476609319 +0000 UTC m=+4.906479159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : secret "metrics-daemon-secret" not found Nov 02 22:48:09.735255 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:09.735220 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5gcjg" event={"ID":"5816c8ac-9af6-4cde-bece-e51d67726b89","Type":"ContainerStarted","Data":"15d55de7a30587f7d24adc06c4cd3580e5c1f87f311a1ca87a722d8c1d4e94cb"} Nov 02 22:48:10.492603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:10.492563 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:10.492603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:10.492606 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:10.492650 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492750 4494 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492770 4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492798 4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492846 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:14.492829904 +0000 UTC m=+8.922699764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492862 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs podName:b586bb58-8500-4dc6-9aef-e1c58f2a39d8 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:14.492856017 +0000 UTC m=+8.922725857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs") pod "network-metrics-daemon-g6zks" (UID: "b586bb58-8500-4dc6-9aef-e1c58f2a39d8") : secret "metrics-daemon-secret" not found Nov 02 22:48:10.492956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:10.492882 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls podName:29ac331f-070d-47b7-9c7b-3fe96663e370 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:14.492866109 +0000 UTC m=+8.922735953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rxg4" (UID: "29ac331f-070d-47b7-9c7b-3fe96663e370") : secret "insights-runtime-extractor-tls" not found Nov 02 22:48:10.628108 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:10.628080 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:11.059584 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.059545 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:11.103496 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.103429 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:11.248901 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.248870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:11.395370 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.395294 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:11.644324 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.644292 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:11.740683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.740659 4494 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:48:11.916480 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:11.916456 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7jpbd_555b9232-5b06-42ff-ae4f-72bf6968ef70/dns-node-resolver/0.log" Nov 02 22:48:12.559688 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:12.559654 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:12.755247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:12.755191 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:12.954739 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:12.954698 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:13.154327 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:13.154293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:13.315815 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:13.315740 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-px8c4_5a94f467-aab7-40b5-a713-356acd286516/node-ca/0.log" Nov 02 22:48:13.354431 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:13.354399 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:13.564947 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:13.564909 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:48:13.916223 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:13.916194 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gt4vq_c738a401-2579-42f8-8876-76c38be04638/serve-healthcheck-canary/0.log" Nov 02 22:48:14.116565 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.116537 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5p6d8_16e628ec-e60c-49da-ac28-34bd947c18d1/migrator/0.log" Nov 02 22:48:14.523079 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.523041 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:14.523307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.523111 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:14.523307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.523137 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:14.523307 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:14.523257 4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 22:48:14.523477 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:14.523322 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls podName:bc97c851-78fa-479b-97cc-7e2b4061342d nodeName:}" failed. No retries permitted until 2026-11-02 22:48:22.523304791 +0000 UTC m=+16.953174631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls") pod "dns-default-t2qbd" (UID: "bc97c851-78fa-479b-97cc-7e2b4061342d") : secret "dns-default-metrics-tls" not found Nov 02 22:48:14.523477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.523446 4494 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:48:14.525687 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.525659 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b586bb58-8500-4dc6-9aef-e1c58f2a39d8-metrics-certs\") pod \"network-metrics-daemon-g6zks\" (UID: \"b586bb58-8500-4dc6-9aef-e1c58f2a39d8\") " pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:14.525800 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.525661 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29ac331f-070d-47b7-9c7b-3fe96663e370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rxg4\" (UID: \"29ac331f-070d-47b7-9c7b-3fe96663e370\") " pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:14.544622 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.544599 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g6zks" Nov 02 22:48:14.549898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.549880 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rxg4" Nov 02 22:48:14.554927 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.554898 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:14.758754 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.758699 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:14.904611 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.904543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:14.984799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:14.984768 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g6zks"] Nov 02 22:48:15.025941 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:15.025914 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb586bb58_8500_4dc6_9aef_e1c58f2a39d8.slice/crio-c2820d54620ed70969f345d53b4daf7443df4aedd64c781161ef71b39084da4b WatchSource:0}: Error finding container c2820d54620ed70969f345d53b4daf7443df4aedd64c781161ef71b39084da4b: Status 404 returned error can't find the container with id c2820d54620ed70969f345d53b4daf7443df4aedd64c781161ef71b39084da4b Nov 02 22:48:15.037252 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.037232 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rxg4"] Nov 02 22:48:15.039623 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:15.039602 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ac331f_070d_47b7_9c7b_3fe96663e370.slice/crio-cee7e49deb71f67bdae88280bb91f6376147be21b5f815ec6d31c2cea8cbb676 WatchSource:0}: Error finding container cee7e49deb71f67bdae88280bb91f6376147be21b5f815ec6d31c2cea8cbb676: Status 404 returned error can't find the container with id cee7e49deb71f67bdae88280bb91f6376147be21b5f815ec6d31c2cea8cbb676 Nov 02 22:48:15.049789 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.049766 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:15.194848 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.194822 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:15.339458 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.339423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:15.482811 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.482725 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:15.752735 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.752655 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rxg4" event={"ID":"29ac331f-070d-47b7-9c7b-3fe96663e370","Type":"ContainerStarted","Data":"5fcf740f0794c9f087504bf558e909fb939069c6949f6b21d33ec6907416f9b1"} Nov 02 22:48:15.752735 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.752713 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rxg4" event={"ID":"29ac331f-070d-47b7-9c7b-3fe96663e370","Type":"ContainerStarted","Data":"cee7e49deb71f67bdae88280bb91f6376147be21b5f815ec6d31c2cea8cbb676"} Nov 02 22:48:15.753936 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.753901 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6zks" event={"ID":"b586bb58-8500-4dc6-9aef-e1c58f2a39d8","Type":"ContainerStarted","Data":"c2820d54620ed70969f345d53b4daf7443df4aedd64c781161ef71b39084da4b"} Nov 02 22:48:15.937311 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:15.937258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:16.361073 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:16.361044 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:16.362424 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:16.362401 4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-179.us-west-1.compute.internal" event="NodeReady" Nov 02 22:48:16.755666 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:16.755624 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:16.956029 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:16.956002 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:17.159117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.158916 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:17.557490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.557456 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:17.704555 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.704516 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:17.762255 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.762217 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rxg4" event={"ID":"29ac331f-070d-47b7-9c7b-3fe96663e370","Type":"ContainerStarted","Data":"e832d8d5b8c8bff17ddaf0077a69ef5bdff598673564e5e3266c495fdb514b4e"} Nov 02 22:48:17.764135 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.764111 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6zks" event={"ID":"b586bb58-8500-4dc6-9aef-e1c58f2a39d8","Type":"ContainerStarted","Data":"85ad5d9484278109d0bb63f7423f611e546b98e4ccd0ce5fda634a5433abbd6e"} Nov 02 22:48:17.764266 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:17.764150 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g6zks" event={"ID":"b586bb58-8500-4dc6-9aef-e1c58f2a39d8","Type":"ContainerStarted","Data":"a46b94f36538c6545b12c2d86e3ee74439283cae2cb131d1217be1a6c73dfcf1"} Nov 02 22:48:18.355862 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:18.355836 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:18.501759 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:18.501729 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:18.757616 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:18.757567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:18.953359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:18.953330 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:19.158165 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:19.158084 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:19.561217 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:19.561158 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:19.771948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:19.771919 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rxg4" event={"ID":"29ac331f-070d-47b7-9c7b-3fe96663e370","Type":"ContainerStarted","Data":"8a1aabbf984d9a0be4d73ab9aab84a18d5b139db182fc1d9fad8db8525922c68"} Nov 02 22:48:20.354835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:20.354796 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:20.758840 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:20.758810 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:20.956764 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:20.956733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:21.150980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:21.150896 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:21.351586 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:21.351547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:22.583770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:22.583732 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:22.586183 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:22.586146 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc97c851-78fa-479b-97cc-7e2b4061342d-metrics-tls\") pod \"dns-default-t2qbd\" (UID: \"bc97c851-78fa-479b-97cc-7e2b4061342d\") " pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:22.647401 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:22.647367 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:23.034850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:23.034823 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2qbd"] Nov 02 22:48:23.037859 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:23.037830 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc97c851_78fa_479b_97cc_7e2b4061342d.slice/crio-56185546092e6d016240a5d8d62eee3e5d80dc767b0bcd7d68870ff7c757a324 WatchSource:0}: Error finding container 56185546092e6d016240a5d8d62eee3e5d80dc767b0bcd7d68870ff7c757a324: Status 404 returned error can't find the container with id 56185546092e6d016240a5d8d62eee3e5d80dc767b0bcd7d68870ff7c757a324 Nov 02 22:48:23.785897 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:23.785850 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2qbd" event={"ID":"bc97c851-78fa-479b-97cc-7e2b4061342d","Type":"ContainerStarted","Data":"56185546092e6d016240a5d8d62eee3e5d80dc767b0bcd7d68870ff7c757a324"} Nov 02 22:48:25.794978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:25.794941 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2qbd" event={"ID":"bc97c851-78fa-479b-97cc-7e2b4061342d","Type":"ContainerStarted","Data":"80c232bc05ee199cc15552f543fcc38de5983831364d99d5e6d2e1dcbb73f879"} Nov 02 22:48:25.795349 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:25.795005 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2qbd" event={"ID":"bc97c851-78fa-479b-97cc-7e2b4061342d","Type":"ContainerStarted","Data":"05cfab7eddd66f003920b5d5f61d5386c1327be10da70c02e1457c1f4001071b"} Nov 02 22:48:25.795349 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:25.795230 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:26.312579 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:26.312540 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:26.455770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:26.455742 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:26.765609 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:26.765580 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:26.854062 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:26.854031 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t92fq" Nov 02 22:48:26.912495 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:26.912465 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:29.860354 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:29.860313 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:30.003712 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:30.003674 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:31.671390 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:31.671353 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:31.828428 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:31.828395 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:31.974711 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:31.974635 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:32.686991 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:32.686965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:32.954824 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:32.954741 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:33.567519 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:33.567493 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9zkr2" Nov 02 22:48:34.121986 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:34.121957 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-792jn" Nov 02 22:48:35.800026 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:35.799996 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2qbd" Nov 02 22:48:35.962660 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:35.962623 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:36.105711 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:36.105634 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:36.249365 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:36.249331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:36.442103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:36.442058 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:36.832351 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:36.832274 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:36.978839 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:36.978811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:37.131683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.131596 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:37.351334 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.351297 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:37.513598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.513569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:37.536100 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.536072 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kcxfk-jfxwl"] Nov 02 22:48:37.538242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.538228 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.585677 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.585656 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.585824 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.585688 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.585824 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.585713 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.585824 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.585730 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg48b\" (UniqueName: \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.596192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.596154 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kcxfk-jfxwl"] Nov 02 22:48:37.659397 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.659370 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:37.686626 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.686599 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.686728 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.686644 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.686779 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.686764 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.686817 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.686795 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg48b\" (UniqueName: \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.687072 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.687052 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.687352 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.687335 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.687468 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.687450 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.748268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.748206 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg48b\" (UniqueName: \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\") pod \"periodic-gathering-kcxfk-jfxwl\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:37.848268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:37.848227 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:48:38.214475 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.214447 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kcxfk-jfxwl"] Nov 02 22:48:38.217911 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:38.217881 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6318b9a_152d_4583_aad6_4c0878a0f111.slice/crio-e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2 WatchSource:0}: Error finding container e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2: Status 404 returned error can't find the container with id e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2 Nov 02 22:48:38.238981 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.238958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:38.387814 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.387781 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:38.836670 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.836637 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" event={"ID":"c6318b9a-152d-4583-aad6-4c0878a0f111","Type":"ContainerStarted","Data":"2396b546cf1ca00b0d5677e9dd7b694ff53074ad6b4478d2678b734900ef00df"} Nov 02 22:48:38.836873 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.836730 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" event={"ID":"c6318b9a-152d-4583-aad6-4c0878a0f111","Type":"ContainerStarted","Data":"e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2"} Nov 02 22:48:38.953147 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:38.953080 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" podStartSLOduration=1.9530657630000001 podStartE2EDuration="1.953065763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:38.952708652 +0000 UTC m=+33.382578513" watchObservedRunningTime="2026-11-02 22:48:38.953065763 +0000 UTC m=+33.382935615" Nov 02 22:48:40.424522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:40.424481 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2qbd_bc97c851-78fa-479b-97cc-7e2b4061342d/dns/0.log" Nov 02 22:48:40.538940 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:40.538911 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2qbd_bc97c851-78fa-479b-97cc-7e2b4061342d/kube-rbac-proxy/0.log" Nov 02 22:48:40.719325 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:40.719257 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7jpbd_555b9232-5b06-42ff-ae4f-72bf6968ef70/dns-node-resolver/0.log" Nov 02 22:48:41.425781 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:41.425743 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:41.720206 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:41.720111 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gt4vq_c738a401-2579-42f8-8876-76c38be04638/serve-healthcheck-canary/0.log" Nov 02 22:48:42.141539 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.141472 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:42.405794 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.405719 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:42.564066 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.564032 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:42.705105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.705071 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:42.753969 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.753924 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp"] Nov 02 22:48:42.760768 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.760738 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.815731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.815704 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:48:42.828002 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.827977 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.828128 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.828010 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.828128 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.828057 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8xw\" (UniqueName: \"kubernetes.io/projected/e281f7d9-8170-49b5-a25a-891bbfa55133-kube-api-access-mb8xw\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.828128 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.828086 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e281f7d9-8170-49b5-a25a-891bbfa55133-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.848880 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.848850 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:42.866273 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.866247 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:48:42.866273 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.866253 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:48:42.866453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.866355 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:48:42.866453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.866362 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2hcn8\"" Nov 02 22:48:42.866453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.866430 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 22:48:42.867024 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.867006 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp"] Nov 02 22:48:42.918518 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.918485 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wtgpq"] Nov 02 22:48:42.920436 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.920420 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:42.928496 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.928475 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8xw\" (UniqueName: \"kubernetes.io/projected/e281f7d9-8170-49b5-a25a-891bbfa55133-kube-api-access-mb8xw\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.928621 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.928510 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e281f7d9-8170-49b5-a25a-891bbfa55133-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.928621 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.928562 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.928621 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.928590 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.929294 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.929274 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e281f7d9-8170-49b5-a25a-891bbfa55133-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.931099 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.931081 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.931320 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.931300 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e281f7d9-8170-49b5-a25a-891bbfa55133-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.989923 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.989859 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8xw\" (UniqueName: \"kubernetes.io/projected/e281f7d9-8170-49b5-a25a-891bbfa55133-kube-api-access-mb8xw\") pod \"openshift-state-metrics-7bd7489957-hgxkp\" (UID: \"e281f7d9-8170-49b5-a25a-891bbfa55133\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:42.993827 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:42.993783 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:43.024005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.023970 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:48:43.024005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.023999 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:48:43.024266 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.024252 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-scv9s\"" Nov 02 22:48:43.024332 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.024270 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:48:43.029373 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029354 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-root\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029385 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-textfile\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029406 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-tls\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029421 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029541 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029469 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ljv\" (UniqueName: \"kubernetes.io/projected/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-kube-api-access-h2ljv\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029626 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-wtmp\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029685 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029657 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-sys\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029685 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029677 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-metrics-client-ca\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.029796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.029718 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.075375 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.075347 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" Nov 02 22:48:43.131012 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.130976 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-metrics-client-ca\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131028 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131071 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-root\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131101 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-textfile\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131132 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-tls\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131187 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131158 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131206 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ljv\" (UniqueName: \"kubernetes.io/projected/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-kube-api-access-h2ljv\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131241 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-wtmp\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131275 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-sys\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131444 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131364 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-sys\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.131979 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.131953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-metrics-client-ca\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.132440 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.132418 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.132513 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.132489 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-root\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.132712 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.132693 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-textfile\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.133289 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.133263 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-wtmp\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.135588 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.135565 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-tls\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.136737 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.136718 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.142304 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.141768 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:43.194670 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.194644 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ljv\" (UniqueName: \"kubernetes.io/projected/d1cb7b72-9689-4ce8-b206-9c39c0c2681c-kube-api-access-h2ljv\") pod \"node-exporter-wtgpq\" (UID: \"d1cb7b72-9689-4ce8-b206-9c39c0c2681c\") " pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.231698 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.231665 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtgpq" Nov 02 22:48:43.241026 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:43.240970 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cb7b72_9689_4ce8_b206_9c39c0c2681c.slice/crio-547e8864cc22efb18b19c33bd7227b03424b47162ac520ebd93582390c77b02b WatchSource:0}: Error finding container 547e8864cc22efb18b19c33bd7227b03424b47162ac520ebd93582390c77b02b: Status 404 returned error can't find the container with id 547e8864cc22efb18b19c33bd7227b03424b47162ac520ebd93582390c77b02b Nov 02 22:48:43.286116 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.286085 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:43.489749 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.489717 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp"] Nov 02 22:48:43.492663 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:43.492608 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281f7d9_8170_49b5_a25a_891bbfa55133.slice/crio-4b0d227c568ac98cead4dfdb9fc5714eccdbc1cce086ff256fb4d59bbce9371b WatchSource:0}: Error finding container 4b0d227c568ac98cead4dfdb9fc5714eccdbc1cce086ff256fb4d59bbce9371b: Status 404 returned error can't find the container with id 4b0d227c568ac98cead4dfdb9fc5714eccdbc1cce086ff256fb4d59bbce9371b Nov 02 22:48:43.853271 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.853189 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtgpq" event={"ID":"d1cb7b72-9689-4ce8-b206-9c39c0c2681c","Type":"ContainerStarted","Data":"547e8864cc22efb18b19c33bd7227b03424b47162ac520ebd93582390c77b02b"} Nov 02 22:48:43.855140 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.855106 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" event={"ID":"e281f7d9-8170-49b5-a25a-891bbfa55133","Type":"ContainerStarted","Data":"23f0c6f840f91baaa344bef9f96855157a134efce5eddabf1268bcab4ca748d9"} Nov 02 22:48:43.855273 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.855164 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" event={"ID":"e281f7d9-8170-49b5-a25a-891bbfa55133","Type":"ContainerStarted","Data":"b9ab0732dfbadb92d47a7336288a0ed615494f35ae35de8069b0d6e1737a3ce4"} Nov 02 22:48:43.855273 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.855213 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" event={"ID":"e281f7d9-8170-49b5-a25a-891bbfa55133","Type":"ContainerStarted","Data":"4b0d227c568ac98cead4dfdb9fc5714eccdbc1cce086ff256fb4d59bbce9371b"} Nov 02 22:48:43.862028 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:43.861997 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.119516 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:44.119434 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.330216 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:44.330167 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.861453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:44.861423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.862674 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:44.862646 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtgpq" event={"ID":"d1cb7b72-9689-4ce8-b206-9c39c0c2681c","Type":"ContainerStarted","Data":"4cde7713f5f77f26bdcc88f25a3a92ab9c850b42a28db446c02044337bac58da"} Nov 02 22:48:45.009487 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.009391 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:45.048844 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.048799 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:48:45.054632 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.054608 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.108417 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108385 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 22:48:45.108543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108402 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 22:48:45.108543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108483 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 22:48:45.108543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108503 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 22:48:45.108543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108525 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 22:48:45.108543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108516 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 22:48:45.108780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108549 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 22:48:45.108780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108614 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 22:48:45.108780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.108731 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-548kw\"" Nov 02 22:48:45.146808 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146776 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.146937 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146815 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.146937 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146833 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.146937 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146856 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146958 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146980 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.146998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147014 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147144 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147066 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147191 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147152 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrx82\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147227 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147191 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147227 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147220 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.147300 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.147278 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.210405 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.210372 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 22:48:45.211504 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.211480 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:48:45.247813 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247781 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrx82\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.247980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247822 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.247980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247845 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.247980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247909 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.247980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247958 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.247988 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248016 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248054 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248092 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248121 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248151 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248236 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248268 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248669 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248647 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.248869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.248813 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.250057 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.250031 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.251246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.251219 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.251246 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.251232 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.251367 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.251336 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.251901 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.251870 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.252098 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.252074 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.252484 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.252461 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.252484 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.252472 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.252894 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.252876 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.253164 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.253148 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.313207 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.313125 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrx82\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\") pod \"alertmanager-main-0\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.369626 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.369594 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:48:45.439241 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.439209 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bb855d47f-vn7b2"] Nov 02 22:48:45.441594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.441579 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.494776 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.494752 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 22:48:45.494932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.494810 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 22:48:45.494932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.494849 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 22:48:45.494932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.494913 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 22:48:45.495263 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.495245 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9vas6hvl3o1s3\"" Nov 02 22:48:45.495328 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.495245 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 22:48:45.495380 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.495367 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8glt8\"" Nov 02 22:48:45.504516 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.504492 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb855d47f-vn7b2"] Nov 02 22:48:45.550589 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550560 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.550589 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550596 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.550837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62tc7\" (UniqueName: \"kubernetes.io/projected/7939c857-e771-4360-a44b-0b0113a23841-kube-api-access-62tc7\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.550837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550696 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.550837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550795 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.550837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550820 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7939c857-e771-4360-a44b-0b0113a23841-tmp\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.551008 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550846 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7939c857-e771-4360-a44b-0b0113a23841-metrics-client-ca\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.551008 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.550867 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.551082 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.551012 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-grpc-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652110 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652029 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652110 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652069 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652110 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652096 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62tc7\" (UniqueName: \"kubernetes.io/projected/7939c857-e771-4360-a44b-0b0113a23841-kube-api-access-62tc7\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652133 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652188 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652212 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7939c857-e771-4360-a44b-0b0113a23841-tmp\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652253 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7939c857-e771-4360-a44b-0b0113a23841-metrics-client-ca\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652283 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652313 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-grpc-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.652702 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.652675 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7939c857-e771-4360-a44b-0b0113a23841-tmp\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.653180 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.653152 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7939c857-e771-4360-a44b-0b0113a23841-metrics-client-ca\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655468 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655444 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655596 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655447 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655596 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655525 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655743 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655725 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655801 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655752 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-grpc-tls\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.655801 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.655762 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7939c857-e771-4360-a44b-0b0113a23841-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.714120 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.714084 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62tc7\" (UniqueName: \"kubernetes.io/projected/7939c857-e771-4360-a44b-0b0113a23841-kube-api-access-62tc7\") pod \"thanos-querier-7bb855d47f-vn7b2\" (UID: \"7939c857-e771-4360-a44b-0b0113a23841\") " pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.756044 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.756014 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:45.795750 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.795488 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:48:45.798108 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:45.798082 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d7eef9_15de_4921_9ccb_88d8c4873305.slice/crio-7b2653042ff297ff5066079c9492207282d23a480b92f58f835525c5c5a30601 WatchSource:0}: Error finding container 7b2653042ff297ff5066079c9492207282d23a480b92f58f835525c5c5a30601: Status 404 returned error can't find the container with id 7b2653042ff297ff5066079c9492207282d23a480b92f58f835525c5c5a30601 Nov 02 22:48:45.869263 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.869221 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"7b2653042ff297ff5066079c9492207282d23a480b92f58f835525c5c5a30601"} Nov 02 22:48:45.870869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.870845 4494 generic.go:356] "Generic (PLEG): container finished" podID="d1cb7b72-9689-4ce8-b206-9c39c0c2681c" containerID="4cde7713f5f77f26bdcc88f25a3a92ab9c850b42a28db446c02044337bac58da" exitCode=0 Nov 02 22:48:45.870990 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.870922 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtgpq" event={"ID":"d1cb7b72-9689-4ce8-b206-9c39c0c2681c","Type":"ContainerDied","Data":"4cde7713f5f77f26bdcc88f25a3a92ab9c850b42a28db446c02044337bac58da"} Nov 02 22:48:45.873031 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:45.873014 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" event={"ID":"e281f7d9-8170-49b5-a25a-891bbfa55133","Type":"ContainerStarted","Data":"a2f76b4196eb354ca430d674e1691f1ae19259158e0f7a4193560e274da7b785"} Nov 02 22:48:46.109089 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.109031 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hgxkp" podStartSLOduration=2.6430733330000002 podStartE2EDuration="4.109015982s" totalImagesPullingTime="1.465942649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:46.10837349 +0000 UTC m=+40.538243351" watchObservedRunningTime="2026-11-02 22:48:46.109015982 +0000 UTC m=+40.538885843" Nov 02 22:48:46.142573 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.142548 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb855d47f-vn7b2"] Nov 02 22:48:46.144116 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:46.144090 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7939c857_e771_4360_a44b_0b0113a23841.slice/crio-ab73046e427dda6ed8d18434471a67318990485df7f4fe1e1236785c5e7cfe3d WatchSource:0}: Error finding container ab73046e427dda6ed8d18434471a67318990485df7f4fe1e1236785c5e7cfe3d: Status 404 returned error can't find the container with id ab73046e427dda6ed8d18434471a67318990485df7f4fe1e1236785c5e7cfe3d Nov 02 22:48:46.879709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.879670 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtgpq" event={"ID":"d1cb7b72-9689-4ce8-b206-9c39c0c2681c","Type":"ContainerStarted","Data":"b732a05fe24e2840e475597a7c46f79621abd9494ef3c1ea1cf19a4ac9aa7d88"} Nov 02 22:48:46.880254 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.879740 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtgpq" event={"ID":"d1cb7b72-9689-4ce8-b206-9c39c0c2681c","Type":"ContainerStarted","Data":"8e79ba3fbb41437b197cb3bc234a4f358d1bf11e7985be1370829b2b839286a8"} Nov 02 22:48:46.881225 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.881200 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"ab73046e427dda6ed8d18434471a67318990485df7f4fe1e1236785c5e7cfe3d"} Nov 02 22:48:46.926022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:46.925985 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.000554 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.000500 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wtgpq" podStartSLOduration=3.412150853 podStartE2EDuration="5.000484505s" totalImagesPullingTime="1.50577891s" totalInitContainerRuntime="82.554742ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:46.999185287 +0000 UTC m=+41.429055148" watchObservedRunningTime="2026-11-02 22:48:47.000484505 +0000 UTC m=+41.430354363" Nov 02 22:48:47.072024 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.071978 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.217393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.217350 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.362692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.362664 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.506478 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.506405 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.564593 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:47.564511 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:48:47.606736 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.606705 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-w6tts"] Nov 02 22:48:47.610041 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.610018 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.652305 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.652272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.667882 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.667852 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 22:48:47.668045 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.668024 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.668045 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.668039 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n7hnx\"" Nov 02 22:48:47.668132 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.668074 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2t4r\" (UniqueName: \"kubernetes.io/projected/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-kube-api-access-v2t4r\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.669036 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.669019 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-w6tts"] Nov 02 22:48:47.759268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.759185 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9f59dcf8c-bxllj"] Nov 02 22:48:47.762504 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.762485 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.768634 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.768565 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.768634 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.768595 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2t4r\" (UniqueName: \"kubernetes.io/projected/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-kube-api-access-v2t4r\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.768930 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:47.768907 4494 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Nov 02 22:48:47.768993 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:48:47.768982 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert podName:f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9 nodeName:}" failed. No retries permitted until 2026-11-02 22:48:48.268961514 +0000 UTC m=+42.698831370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-w6tts" (UID: "f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9") : secret "monitoring-plugin-cert" not found Nov 02 22:48:47.815546 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815432 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 22:48:47.815709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815582 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 22:48:47.815709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815679 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:48:47.815709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815695 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cepd418fea7sv\"" Nov 02 22:48:47.815709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815692 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 22:48:47.815709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.815739 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hlfgm\"" Nov 02 22:48:47.821383 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.821357 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9f59dcf8c-bxllj"] Nov 02 22:48:47.829851 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.829828 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2t4r\" (UniqueName: \"kubernetes.io/projected/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-kube-api-access-v2t4r\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:47.869836 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.869793 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870016 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.869918 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxtt\" (UniqueName: \"kubernetes.io/projected/c9613f21-6e36-47cc-9633-c2865cd76577-kube-api-access-5zxtt\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870016 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.869970 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-client-certs\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870016 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.869994 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-tls\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870196 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.870027 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9613f21-6e36-47cc-9633-c2865cd76577-audit-log\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870196 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.870137 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-metrics-server-audit-profiles\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.870303 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.870233 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-client-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.887191 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.887135 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5" exitCode=0 Nov 02 22:48:47.887622 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.887228 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5"} Nov 02 22:48:47.903514 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.903473 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.970893 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.970859 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9613f21-6e36-47cc-9633-c2865cd76577-audit-log\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971078 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.970918 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-metrics-server-audit-profiles\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971078 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.970968 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-client-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971078 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.971074 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971288 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.971214 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxtt\" (UniqueName: \"kubernetes.io/projected/c9613f21-6e36-47cc-9633-c2865cd76577-kube-api-access-5zxtt\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971350 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.971332 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-client-certs\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971403 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.971366 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-tls\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.971403 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.971367 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c9613f21-6e36-47cc-9633-c2865cd76577-audit-log\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.972076 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.972046 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.972287 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.972264 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c9613f21-6e36-47cc-9633-c2865cd76577-metrics-server-audit-profiles\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.974196 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.974141 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-tls\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.974483 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.974465 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-client-ca-bundle\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:47.974530 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:47.974516 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c9613f21-6e36-47cc-9633-c2865cd76577-secret-metrics-server-client-certs\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:48.036104 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.036026 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxtt\" (UniqueName: \"kubernetes.io/projected/c9613f21-6e36-47cc-9633-c2865cd76577-kube-api-access-5zxtt\") pod \"metrics-server-9f59dcf8c-bxllj\" (UID: \"c9613f21-6e36-47cc-9633-c2865cd76577\") " pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:48.072985 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.072953 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:48:48.274018 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.273965 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:48.276926 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.276898 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-w6tts\" (UID: \"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:48.497802 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.497767 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9f59dcf8c-bxllj"] Nov 02 22:48:48.501889 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:48.501856 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9613f21_6e36_47cc_9633_c2865cd76577.slice/crio-86a11c4a44bf6536bedf439684d46b597c32d1057f59ef77b419b4f443d03217 WatchSource:0}: Error finding container 86a11c4a44bf6536bedf439684d46b597c32d1057f59ef77b419b4f443d03217: Status 404 returned error can't find the container with id 86a11c4a44bf6536bedf439684d46b597c32d1057f59ef77b419b4f443d03217 Nov 02 22:48:48.522978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.522956 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:48.892127 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.892094 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" event={"ID":"c9613f21-6e36-47cc-9633-c2865cd76577","Type":"ContainerStarted","Data":"86a11c4a44bf6536bedf439684d46b597c32d1057f59ef77b419b4f443d03217"} Nov 02 22:48:48.894209 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.894184 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"f0de61ee8ecb7a90d5e919c1004d027ed08aa38c6bf7370e745dbe029604ee49"} Nov 02 22:48:48.894262 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.894249 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"0f599023af61d16d79fab5249d070bdbf4b17663f617559c5bd4da6da4d98806"} Nov 02 22:48:48.894294 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.894282 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"fe333c7bfd553241fa8caaf8ebb2804ca13ff69fb88cacd9a86fd65b59725ef3"} Nov 02 22:48:48.900490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:48.900476 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-w6tts"] Nov 02 22:48:48.902861 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:48.902835 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e6242e_8838_4cc2_b1ef_62e1a3f78bb9.slice/crio-83e1fdb7b23eea1cba3b09c1d4b95767a4561737d60a0362db10accb8dfbfe04 WatchSource:0}: Error finding container 83e1fdb7b23eea1cba3b09c1d4b95767a4561737d60a0362db10accb8dfbfe04: Status 404 returned error can't find the container with id 83e1fdb7b23eea1cba3b09c1d4b95767a4561737d60a0362db10accb8dfbfe04 Nov 02 22:48:49.031738 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.031664 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:48:49.035138 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.035107 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.088746 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088720 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:48:49.088746 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088729 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:48:49.088746 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088754 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:48:49.089020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088809 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:48:49.089020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088918 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:48:49.089020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088953 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:48:49.089020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.088979 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kplp2\"" Nov 02 22:48:49.089156 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.089143 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:48:49.095455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.095432 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:48:49.183587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183555 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183587 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183588 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183818 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183622 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183818 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183729 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183818 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183818 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183777 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jww\" (UniqueName: \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.183982 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.183845 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.242354 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.242317 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:49.284603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284535 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284576 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jww\" (UniqueName: \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284620 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284672 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284697 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284747 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.284835 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.284786 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.285075 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.285016 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.285402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.285377 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.285402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.285389 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.285641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.285617 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.287854 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.287834 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.287942 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.287838 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.346768 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.346739 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jww\" (UniqueName: \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\") pod \"console-7ff9bb5946-md9fr\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.349448 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.349425 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:49.383541 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.383507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:49.738471 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.738402 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:48:49.741153 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:49.741122 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf020545d_c867_455e_9fa0_d2ef840419c2.slice/crio-b98f23e36ec6bd82fc529925730d210d11647f3295e4306b62789657da78e7ef WatchSource:0}: Error finding container b98f23e36ec6bd82fc529925730d210d11647f3295e4306b62789657da78e7ef: Status 404 returned error can't find the container with id b98f23e36ec6bd82fc529925730d210d11647f3295e4306b62789657da78e7ef Nov 02 22:48:49.904297 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.904258 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff9bb5946-md9fr" event={"ID":"f020545d-c867-455e-9fa0-d2ef840419c2","Type":"ContainerStarted","Data":"b98f23e36ec6bd82fc529925730d210d11647f3295e4306b62789657da78e7ef"} Nov 02 22:48:49.906339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:49.906307 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" event={"ID":"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9","Type":"ContainerStarted","Data":"83e1fdb7b23eea1cba3b09c1d4b95767a4561737d60a0362db10accb8dfbfe04"} Nov 02 22:48:50.916339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f"} Nov 02 22:48:50.916780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916413 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109"} Nov 02 22:48:50.916780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916455 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef"} Nov 02 22:48:50.916780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916495 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944"} Nov 02 22:48:50.916780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916535 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707"} Nov 02 22:48:50.916780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.916576 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerStarted","Data":"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737"} Nov 02 22:48:50.918150 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.918117 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" event={"ID":"c9613f21-6e36-47cc-9633-c2865cd76577","Type":"ContainerStarted","Data":"c69d689119d78bb25bb51e8083137bed62dc0e77581fd0720e3c2491bb30a50c"} Nov 02 22:48:50.923789 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.923632 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"801ce454867e80b6e07c1220b300868c7f736fb34464a724b5e1f78d75fbca60"} Nov 02 22:48:50.923789 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.923706 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"dd4f49e277c2cd673ea6a78f9c2f4dea694d5f128b4b6fe1a981676e68ac36d7"} Nov 02 22:48:50.923789 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.923751 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" event={"ID":"7939c857-e771-4360-a44b-0b0113a23841","Type":"ContainerStarted","Data":"d0cd75f788a6d4f3bb9a8ffc0646543442d177e5f831fcf266fef97a5d7a3694"} Nov 02 22:48:50.923997 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:50.923875 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:51.131887 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.131830 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.331609421 podStartE2EDuration="6.131809856s" totalImagesPullingTime="3.785875528s" totalInitContainerRuntime="14.324907ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:51.09705773 +0000 UTC m=+45.526927592" watchObservedRunningTime="2026-11-02 22:48:51.131809856 +0000 UTC m=+45.561679717" Nov 02 22:48:51.226357 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.225153 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" podStartSLOduration=2.294112 podStartE2EDuration="6.225119061s" totalImagesPullingTime="3.931007061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:51.224780096 +0000 UTC m=+45.654649958" watchObservedRunningTime="2026-11-02 22:48:51.225119061 +0000 UTC m=+45.654988924" Nov 02 22:48:51.344101 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.344039 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" podStartSLOduration=2.255879246 podStartE2EDuration="4.344021055s" totalImagesPullingTime="2.088141809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:51.343361957 +0000 UTC m=+45.773231817" watchObservedRunningTime="2026-11-02 22:48:51.344021055 +0000 UTC m=+45.773890918" Nov 02 22:48:51.832547 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.832509 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:51.868867 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.868832 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:48:51.873067 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.873045 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:51.926907 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.926874 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:48:51.926907 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.926972 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:48:51.926907 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.926985 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927132 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8cslm08d5q9bp\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927146 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927166 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927229 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927229 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927263 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927299 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927327 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927484 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927544 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:48:51.927601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.927549 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xcgqq\"" Nov 02 22:48:51.929930 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.929839 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" event={"ID":"f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9","Type":"ContainerStarted","Data":"95507a02708b8eb4dfcb2a206799eb3959cfc6775ca4bbf630540d581c178a30"} Nov 02 22:48:51.930474 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.930453 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:51.936084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.936050 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" Nov 02 22:48:51.978729 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.978591 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:48:51.978729 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.978674 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:48:51.981871 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:51.981845 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:48:52.008510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.008477 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.008640 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.008513 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.008640 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.008605 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.008885 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.008863 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.009209 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.009162 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.009575 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.009555 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.009954 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.009927 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010070 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010013 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010070 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010043 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvm8m\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010199 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010124 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010199 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010149 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010199 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010195 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010351 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010259 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010351 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010286 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010351 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010309 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010496 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010398 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010496 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010428 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010639 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010528 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010639 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010622 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.010734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.010647 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111523 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111574 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111611 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111649 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111675 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111801 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.111850 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111841 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111871 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111918 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111964 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.111990 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112024 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112087 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112028 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112132 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112162 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvm8m\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112231 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112259 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112288 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112345 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.112409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.112369 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.114045 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.113237 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.114045 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.113313 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.114045 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.113932 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.116948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.115220 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.116948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.115427 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.116948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.116778 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.116948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.116789 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.116948 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.116911 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.118012 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.117209 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.118012 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.117958 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.119692 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.119705 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.119779 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.120018 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.120078 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.120095 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120591 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.120570 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.120837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.120808 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.145166 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.145121 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-w6tts" podStartSLOduration=2.974861716 podStartE2EDuration="5.145107567s" totalImagesPullingTime="2.170245851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:52.144330967 +0000 UTC m=+46.574200827" watchObservedRunningTime="2026-11-02 22:48:52.145107567 +0000 UTC m=+46.574977429" Nov 02 22:48:52.178709 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.178677 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvm8m\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\") pod \"prometheus-k8s-0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.200397 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.200365 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:52.638603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.638499 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:48:52.641335 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:48:52.641310 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456d1d47_09a8_4217_98f0_613cf8494ea0.slice/crio-80a0ef00a8d03de7618305c9284e76132d5a1bda63dce7ed0b2ea2e4f5599572 WatchSource:0}: Error finding container 80a0ef00a8d03de7618305c9284e76132d5a1bda63dce7ed0b2ea2e4f5599572: Status 404 returned error can't find the container with id 80a0ef00a8d03de7618305c9284e76132d5a1bda63dce7ed0b2ea2e4f5599572 Nov 02 22:48:52.935898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.935859 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" exitCode=0 Nov 02 22:48:52.936512 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.935948 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} Nov 02 22:48:52.936512 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:52.936086 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"80a0ef00a8d03de7618305c9284e76132d5a1bda63dce7ed0b2ea2e4f5599572"} Nov 02 22:48:53.634893 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:53.634813 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:53.782387 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:53.782351 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:53.943286 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:53.943242 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff9bb5946-md9fr" event={"ID":"f020545d-c867-455e-9fa0-d2ef840419c2","Type":"ContainerStarted","Data":"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca"} Nov 02 22:48:54.047422 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.047389 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:54.062402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.062349 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7ff9bb5946-md9fr" podStartSLOduration=2.6664167 podStartE2EDuration="6.062332604s" totalImagesPullingTime="3.395915904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:54.062061943 +0000 UTC m=+48.491931804" watchObservedRunningTime="2026-11-02 22:48:54.062332604 +0000 UTC m=+48.492202465" Nov 02 22:48:54.190556 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.190519 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:54.501016 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.500978 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:54.646845 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.646805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:54.789339 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.789253 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:54.935541 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:54.935507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:55.077153 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:55.077054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:55.352324 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:55.352227 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:55.522442 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:55.522405 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:56.938140 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.938115 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bb855d47f-vn7b2" Nov 02 22:48:56.958790 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958758 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} Nov 02 22:48:56.958918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958815 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} Nov 02 22:48:56.958918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958840 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} Nov 02 22:48:56.958918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958865 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} Nov 02 22:48:56.958918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958893 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} Nov 02 22:48:56.958918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:56.958915 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerStarted","Data":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} Nov 02 22:48:57.201096 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.201050 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:48:57.300521 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.300469 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.684593442 podStartE2EDuration="6.300453014s" totalImagesPullingTime="3.595968828s" totalInitContainerRuntime="19.890744ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:57.299708601 +0000 UTC m=+51.729578463" watchObservedRunningTime="2026-11-02 22:48:57.300453014 +0000 UTC m=+51.730322876" Nov 02 22:48:57.819541 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.819507 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:57.819822 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.819780 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" podUID="d6144b49-0219-4f29-9470-64494e6d08b8" containerName="insights-gathering" containerID="cri-o://db4d62647fff59c6125fcbda74718d1f4575adb0fcaac9615a81ec74ebaf4b46" gracePeriod=30 Nov 02 22:48:57.964955 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.964918 4494 generic.go:356] "Generic (PLEG): container finished" podID="d6144b49-0219-4f29-9470-64494e6d08b8" containerID="db4d62647fff59c6125fcbda74718d1f4575adb0fcaac9615a81ec74ebaf4b46" exitCode=2 Nov 02 22:48:57.965457 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:57.965001 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" event={"ID":"d6144b49-0219-4f29-9470-64494e6d08b8","Type":"ContainerDied","Data":"db4d62647fff59c6125fcbda74718d1f4575adb0fcaac9615a81ec74ebaf4b46"} Nov 02 22:48:58.062273 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.062246 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:58.174895 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.174864 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") pod \"d6144b49-0219-4f29-9470-64494e6d08b8\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " Nov 02 22:48:58.175055 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.174927 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") pod \"d6144b49-0219-4f29-9470-64494e6d08b8\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " Nov 02 22:48:58.175055 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.174957 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") pod \"d6144b49-0219-4f29-9470-64494e6d08b8\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " Nov 02 22:48:58.175055 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.175000 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") pod \"d6144b49-0219-4f29-9470-64494e6d08b8\" (UID: \"d6144b49-0219-4f29-9470-64494e6d08b8\") " Nov 02 22:48:58.175352 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.175327 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path" pod "d6144b49-0219-4f29-9470-64494e6d08b8" (UID: "d6144b49-0219-4f29-9470-64494e6d08b8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:48:58.175400 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.175353 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle" pod "d6144b49-0219-4f29-9470-64494e6d08b8" (UID: "d6144b49-0219-4f29-9470-64494e6d08b8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:58.175459 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.175441 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle" pod "d6144b49-0219-4f29-9470-64494e6d08b8" (UID: "d6144b49-0219-4f29-9470-64494e6d08b8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:58.177476 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.177453 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn" pod "d6144b49-0219-4f29-9470-64494e6d08b8" (UID: "d6144b49-0219-4f29-9470-64494e6d08b8"). InnerVolumeSpecName "kube-api-access-dmwkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:58.276561 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.276521 4494 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d6144b49-0219-4f29-9470-64494e6d08b8-archives-path\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:58.276561 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.276554 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmwkn\" (UniqueName: \"kubernetes.io/projected/d6144b49-0219-4f29-9470-64494e6d08b8-kube-api-access-dmwkn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:58.276561 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.276570 4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-trusted-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:58.276799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.276583 4494 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6144b49-0219-4f29-9470-64494e6d08b8-service-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:48:58.970403 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.970362 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" event={"ID":"d6144b49-0219-4f29-9470-64494e6d08b8","Type":"ContainerDied","Data":"991ef4f3dd84027027d919668c3fb8c8ed4cf186aced4c402564e822f051d921"} Nov 02 22:48:58.970823 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.970431 4494 scope.go:122] "RemoveContainer" containerID="db4d62647fff59c6125fcbda74718d1f4575adb0fcaac9615a81ec74ebaf4b46" Nov 02 22:48:58.970823 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:58.970380 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rjlhh-4swj9" Nov 02 22:48:59.040418 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.040384 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:59.135231 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.135201 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:59.139089 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.139067 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rjlhh-4swj9"] Nov 02 22:48:59.186495 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.186461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:59.349957 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.349863 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:59.350127 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.349964 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:59.355166 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.355145 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:48:59.690859 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.690822 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6144b49-0219-4f29-9470-64494e6d08b8" path="/var/lib/kubelet/pods/d6144b49-0219-4f29-9470-64494e6d08b8/volumes" Nov 02 22:48:59.980192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:48:59.980102 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:49:06.943584 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:06.943550 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:06.962733 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:06.962706 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:07.024639 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:07.024603 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:08.073194 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:08.073143 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:49:08.073682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:08.073464 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:49:11.155936 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:11.155904 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:49:11.217748 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:11.217721 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:11.367668 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:11.367638 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:12.938767 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:12.938736 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:13.084491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:13.084458 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:14.865586 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:14.865552 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:15.010309 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:15.010277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:17.557468 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:17.557433 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:49:17.648377 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:17.648346 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:17.790777 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:17.790740 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:25.540963 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:25.540922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:25.689598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:25.689543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:28.079089 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:28.079054 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:49:28.083005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:28.082982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9f59dcf8c-bxllj" Nov 02 22:49:29.910498 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:29.910462 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:30.113389 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:30.113355 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:34.378552 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:34.378509 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:34.617891 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:34.617855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:36.176139 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.176098 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7ff9bb5946-md9fr" podUID="f020545d-c867-455e-9fa0-d2ef840419c2" containerName="console" containerID="cri-o://fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca" gracePeriod=15 Nov 02 22:49:36.421806 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.421780 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:49:36.511111 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511043 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511111 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511105 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511125 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\" (UniqueName: \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511149 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511208 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511237 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511321 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511259 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\") pod \"f020545d-c867-455e-9fa0-d2ef840419c2\" (UID: \"f020545d-c867-455e-9fa0-d2ef840419c2\") " Nov 02 22:49:36.511621 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511571 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:36.511730 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511674 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:36.511730 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511716 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:36.511840 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.511726 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:36.513567 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.513543 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "kube-api-access-p2jww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:36.513972 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.513945 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:36.514057 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.513954 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config" pod "f020545d-c867-455e-9fa0-d2ef840419c2" (UID: "f020545d-c867-455e-9fa0-d2ef840419c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:36.582644 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.582606 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:36.583204 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583132 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="alertmanager" containerID="cri-o://44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737" gracePeriod=120 Nov 02 22:49:36.583454 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583411 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy" containerID="cri-o://d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef" gracePeriod=120 Nov 02 22:49:36.583598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583512 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-web" containerID="cri-o://a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944" gracePeriod=120 Nov 02 22:49:36.583598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583575 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="prom-label-proxy" containerID="cri-o://1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f" gracePeriod=120 Nov 02 22:49:36.583775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583629 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="config-reloader" containerID="cri-o://cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707" gracePeriod=120 Nov 02 22:49:36.583775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.583684 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-metric" containerID="cri-o://89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109" gracePeriod=120 Nov 02 22:49:36.612386 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612363 4494 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-console-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612386 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612387 4494 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612397 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jww\" (UniqueName: \"kubernetes.io/projected/f020545d-c867-455e-9fa0-d2ef840419c2-kube-api-access-p2jww\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612408 4494 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-serving-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612416 4494 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f020545d-c867-455e-9fa0-d2ef840419c2-console-oauth-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612424 4494 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f020545d-c867-455e-9fa0-d2ef840419c2-oauth-serving-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:36.612506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:36.612433 4494 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f020545d-c867-455e-9fa0-d2ef840419c2-tmp\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:37.116149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116111 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f" exitCode=0 Nov 02 22:49:37.116149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116135 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109" exitCode=0 Nov 02 22:49:37.116149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116140 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef" exitCode=0 Nov 02 22:49:37.116149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116146 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707" exitCode=0 Nov 02 22:49:37.116149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116151 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737" exitCode=0 Nov 02 22:49:37.116477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116196 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f"} Nov 02 22:49:37.116477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116270 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109"} Nov 02 22:49:37.116477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116310 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef"} Nov 02 22:49:37.116477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116340 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707"} Nov 02 22:49:37.116477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.116364 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737"} Nov 02 22:49:37.117491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.117465 4494 generic.go:356] "Generic (PLEG): container finished" podID="f020545d-c867-455e-9fa0-d2ef840419c2" containerID="fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca" exitCode=0 Nov 02 22:49:37.117545 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.117495 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff9bb5946-md9fr" event={"ID":"f020545d-c867-455e-9fa0-d2ef840419c2","Type":"ContainerDied","Data":"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca"} Nov 02 22:49:37.117545 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.117525 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff9bb5946-md9fr" event={"ID":"f020545d-c867-455e-9fa0-d2ef840419c2","Type":"ContainerDied","Data":"b98f23e36ec6bd82fc529925730d210d11647f3295e4306b62789657da78e7ef"} Nov 02 22:49:37.117545 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.117526 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff9bb5946-md9fr" Nov 02 22:49:37.117630 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.117538 4494 scope.go:122] "RemoveContainer" containerID="fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca" Nov 02 22:49:37.128149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.128132 4494 scope.go:122] "RemoveContainer" containerID="fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca" Nov 02 22:49:37.128469 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:37.128442 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca\": container with ID starting with fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca not found: ID does not exist" containerID="fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca" Nov 02 22:49:37.128538 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.128477 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca"} err="failed to get container status \"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca\": rpc error: code = NotFound desc = could not find container \"fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca\": container with ID starting with fd0f0262c5de4f006078931a339f4726720aaaf6a74280ae8057a9b7252918ca not found: ID does not exist" Nov 02 22:49:37.284924 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.284870 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:49:37.286327 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.286308 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ff9bb5946-md9fr"] Nov 02 22:49:37.690470 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.690439 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f020545d-c867-455e-9fa0-d2ef840419c2" path="/var/lib/kubelet/pods/f020545d-c867-455e-9fa0-d2ef840419c2/volumes" Nov 02 22:49:37.832455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.832433 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:37.921475 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921445 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921492 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921512 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921552 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921578 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921597 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921661 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921630 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921666 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921711 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921744 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921783 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921823 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.921976 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.921862 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\") pod \"c8d7eef9-15de-4921-9ccb-88d8c4873305\" (UID: \"c8d7eef9-15de-4921-9ccb-88d8c4873305\") " Nov 02 22:49:37.922468 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.922007 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:37.922468 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.922275 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:37.922468 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.922413 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-metrics-client-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:37.923287 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.923041 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:37.925125 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925085 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.925425 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925396 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:37.925799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925626 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "kube-api-access-jrx82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:37.925799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925724 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.925799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925750 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.925799 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.925779 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.926504 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.926474 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.926571 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.926540 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:37.930571 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.930453 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:37.936757 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:37.936733 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config" pod "c8d7eef9-15de-4921-9ccb-88d8c4873305" (UID: "c8d7eef9-15de-4921-9ccb-88d8c4873305"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:38.023331 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023294 4494 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023331 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023324 4494 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-web-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023331 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023335 4494 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-main-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023352 4494 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-main-db\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023361 4494 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d7eef9-15de-4921-9ccb-88d8c4873305-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023370 4494 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-out\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023378 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrx82\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-kube-api-access-jrx82\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023386 4494 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023394 4494 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-cluster-tls-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023403 4494 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023411 4494 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d7eef9-15de-4921-9ccb-88d8c4873305-tls-assets\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.023522 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.023423 4494 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d7eef9-15de-4921-9ccb-88d8c4873305-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:38.125338 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.125302 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerID="a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944" exitCode=0 Nov 02 22:49:38.125493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.125383 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944"} Nov 02 22:49:38.125493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.125395 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.125493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.125478 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8d7eef9-15de-4921-9ccb-88d8c4873305","Type":"ContainerDied","Data":"7b2653042ff297ff5066079c9492207282d23a480b92f58f835525c5c5a30601"} Nov 02 22:49:38.125648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.125531 4494 scope.go:122] "RemoveContainer" containerID="1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f" Nov 02 22:49:38.134325 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.134160 4494 scope.go:122] "RemoveContainer" containerID="89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109" Nov 02 22:49:38.141703 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.141688 4494 scope.go:122] "RemoveContainer" containerID="d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef" Nov 02 22:49:38.148531 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.148510 4494 scope.go:122] "RemoveContainer" containerID="a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944" Nov 02 22:49:38.155118 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.155098 4494 scope.go:122] "RemoveContainer" containerID="cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707" Nov 02 22:49:38.163269 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.163242 4494 scope.go:122] "RemoveContainer" containerID="44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737" Nov 02 22:49:38.170431 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.170402 4494 scope.go:122] "RemoveContainer" containerID="8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5" Nov 02 22:49:38.177407 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.177389 4494 scope.go:122] "RemoveContainer" containerID="1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f" Nov 02 22:49:38.177671 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.177652 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f\": container with ID starting with 1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f not found: ID does not exist" containerID="1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f" Nov 02 22:49:38.177716 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.177684 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f"} err="failed to get container status \"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f\": rpc error: code = NotFound desc = could not find container \"1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f\": container with ID starting with 1630ddf1326d0b73933a176df8c8f38a76959871960cc5f8cedd6365abcd132f not found: ID does not exist" Nov 02 22:49:38.177716 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.177704 4494 scope.go:122] "RemoveContainer" containerID="89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109" Nov 02 22:49:38.177955 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.177937 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109\": container with ID starting with 89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109 not found: ID does not exist" containerID="89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109" Nov 02 22:49:38.178023 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.177965 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109"} err="failed to get container status \"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109\": rpc error: code = NotFound desc = could not find container \"89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109\": container with ID starting with 89519e0d0ded20ec82c2f2417ff06efcf391192d6e5a67c60eb17e4c6d851109 not found: ID does not exist" Nov 02 22:49:38.178023 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.177995 4494 scope.go:122] "RemoveContainer" containerID="d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef" Nov 02 22:49:38.178251 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.178233 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef\": container with ID starting with d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef not found: ID does not exist" containerID="d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef" Nov 02 22:49:38.178307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178257 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef"} err="failed to get container status \"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef\": rpc error: code = NotFound desc = could not find container \"d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef\": container with ID starting with d752d5d9f1a535a09a8cd0cba5242a6216039ea4053f300148f20a4e0b9b94ef not found: ID does not exist" Nov 02 22:49:38.178307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178275 4494 scope.go:122] "RemoveContainer" containerID="a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944" Nov 02 22:49:38.178494 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.178479 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944\": container with ID starting with a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944 not found: ID does not exist" containerID="a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944" Nov 02 22:49:38.178533 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178497 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944"} err="failed to get container status \"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944\": rpc error: code = NotFound desc = could not find container \"a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944\": container with ID starting with a6f784678e971adf1c25f388c36d903f993132d106c73fce122a92c6189f7944 not found: ID does not exist" Nov 02 22:49:38.178533 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178510 4494 scope.go:122] "RemoveContainer" containerID="cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707" Nov 02 22:49:38.178731 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.178694 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707\": container with ID starting with cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707 not found: ID does not exist" containerID="cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707" Nov 02 22:49:38.178775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178736 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707"} err="failed to get container status \"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707\": rpc error: code = NotFound desc = could not find container \"cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707\": container with ID starting with cb5088d6e1f50907c20fe51ff8c64891e7bdb08e3b4000d437b784e273281707 not found: ID does not exist" Nov 02 22:49:38.178775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178748 4494 scope.go:122] "RemoveContainer" containerID="44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737" Nov 02 22:49:38.178947 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.178930 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737\": container with ID starting with 44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737 not found: ID does not exist" containerID="44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737" Nov 02 22:49:38.178986 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178954 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737"} err="failed to get container status \"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737\": rpc error: code = NotFound desc = could not find container \"44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737\": container with ID starting with 44dae30002eb08d5a2d73a6a11374318b9368f566d252c98cdbe427d784a9737 not found: ID does not exist" Nov 02 22:49:38.178986 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.178968 4494 scope.go:122] "RemoveContainer" containerID="8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5" Nov 02 22:49:38.179195 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:38.179159 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5\": container with ID starting with 8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5 not found: ID does not exist" containerID="8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5" Nov 02 22:49:38.179245 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.179202 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5"} err="failed to get container status \"8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5\": rpc error: code = NotFound desc = could not find container \"8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5\": container with ID starting with 8e7edfcdab5877d2734d94c280b4be0975d9dc0ba4d81f56acc576d68877aca5 not found: ID does not exist" Nov 02 22:49:38.294373 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.294332 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:38.298463 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.298437 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:38.344589 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344561 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:38.344845 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344834 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="init-config-reloader" Nov 02 22:49:38.344884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344846 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="init-config-reloader" Nov 02 22:49:38.344884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344855 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="prom-label-proxy" Nov 02 22:49:38.344884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344860 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="prom-label-proxy" Nov 02 22:49:38.344884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344872 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-metric" Nov 02 22:49:38.344884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344877 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-metric" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344888 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="config-reloader" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344893 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="config-reloader" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344901 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f020545d-c867-455e-9fa0-d2ef840419c2" containerName="console" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344905 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f020545d-c867-455e-9fa0-d2ef840419c2" containerName="console" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344917 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d6144b49-0219-4f29-9470-64494e6d08b8" containerName="insights-gathering" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344921 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6144b49-0219-4f29-9470-64494e6d08b8" containerName="insights-gathering" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344928 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-web" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344932 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-web" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344942 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="alertmanager" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344947 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="alertmanager" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344953 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344957 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.344996 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-metric" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345004 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="config-reloader" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345009 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f020545d-c867-455e-9fa0-d2ef840419c2" containerName="console" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345015 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6144b49-0219-4f29-9470-64494e6d08b8" containerName="insights-gathering" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345020 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy-web" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345026 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="alertmanager" Nov 02 22:49:38.345025 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345030 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="kube-rbac-proxy" Nov 02 22:49:38.345570 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.345036 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" containerName="prom-label-proxy" Nov 02 22:49:38.348491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.348474 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.402086 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402043 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 22:49:38.402086 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402043 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 22:49:38.402086 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402043 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 22:49:38.402086 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402059 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 22:49:38.402419 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402128 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 22:49:38.402419 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402142 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 22:49:38.402419 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402144 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 22:49:38.402419 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402283 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 22:49:38.402610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.402497 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-548kw\"" Nov 02 22:49:38.426441 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426410 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426613 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426461 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426613 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426698 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426641 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426736 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426697 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426736 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426725 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426806 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426749 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-config-out\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426806 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426767 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426806 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426805 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-web-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426953 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426824 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426953 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426840 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkr8\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-kube-api-access-knkr8\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426953 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426864 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-config-volume\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.426953 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.426878 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.452680 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.452617 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 22:49:38.453867 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.453847 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:38.527562 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527533 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527572 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527611 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527641 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527675 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-config-out\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527700 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527750 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-web-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527777 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527798 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkr8\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-kube-api-access-knkr8\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527827 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-config-volume\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527849 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527876 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.527960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.527908 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.528424 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.528377 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.529241 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.529216 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.529714 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.529658 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26470e00-2773-4d56-b168-b3ce323c57df-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.531009 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.530981 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.531009 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.530997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.531585 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.531563 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-web-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.531565 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.532001 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-config-volume\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.532001 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.532043 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.532076 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26470e00-2773-4d56-b168-b3ce323c57df-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.532920 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.532902 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26470e00-2773-4d56-b168-b3ce323c57df-config-out\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.590548 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.590522 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkr8\" (UniqueName: \"kubernetes.io/projected/26470e00-2773-4d56-b168-b3ce323c57df-kube-api-access-knkr8\") pod \"alertmanager-main-0\" (UID: \"26470e00-2773-4d56-b168-b3ce323c57df\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:38.663376 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:38.663340 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:49:39.092624 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:39.092599 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:49:39.094855 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:49:39.094828 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26470e00_2773_4d56_b168_b3ce323c57df.slice/crio-e5e9c810109bf6965bd8c5470f4fab06fdab7dd93f223f0c962d51827d11d47d WatchSource:0}: Error finding container e5e9c810109bf6965bd8c5470f4fab06fdab7dd93f223f0c962d51827d11d47d: Status 404 returned error can't find the container with id e5e9c810109bf6965bd8c5470f4fab06fdab7dd93f223f0c962d51827d11d47d Nov 02 22:49:39.132793 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:39.132758 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"e5e9c810109bf6965bd8c5470f4fab06fdab7dd93f223f0c962d51827d11d47d"} Nov 02 22:49:39.690614 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:39.690585 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d7eef9-15de-4921-9ccb-88d8c4873305" path="/var/lib/kubelet/pods/c8d7eef9-15de-4921-9ccb-88d8c4873305/volumes" Nov 02 22:49:40.138092 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.138011 4494 generic.go:356] "Generic (PLEG): container finished" podID="26470e00-2773-4d56-b168-b3ce323c57df" containerID="a407ebee0d0dc0431dc7e3177f712c224798da970962d133d5ee8101bfafb045" exitCode=0 Nov 02 22:49:40.138092 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.138057 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerDied","Data":"a407ebee0d0dc0431dc7e3177f712c224798da970962d133d5ee8101bfafb045"} Nov 02 22:49:40.260518 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.260493 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f595c5795-zfd8j"] Nov 02 22:49:40.263558 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.263539 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.318937 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.318899 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5bv6b\"" Nov 02 22:49:40.319046 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.319032 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 22:49:40.342641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342614 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-serving-certs-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.342761 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342648 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.342761 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342683 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-federate-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.342761 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342737 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-metrics-client-ca\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.342888 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342849 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.342992 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.342973 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.343056 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.343009 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.343056 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.343041 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5nv\" (UniqueName: \"kubernetes.io/projected/866403c0-bfc0-4e67-8424-1686b798b6ba-kube-api-access-2q5nv\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.369713 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.369690 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 22:49:40.421142 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.421109 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 22:49:40.421278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.421250 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 22:49:40.421278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.421264 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 22:49:40.421440 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.421421 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 22:49:40.423345 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.423323 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f595c5795-zfd8j"] Nov 02 22:49:40.443604 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443578 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443701 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443611 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443701 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443634 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5nv\" (UniqueName: \"kubernetes.io/projected/866403c0-bfc0-4e67-8424-1686b798b6ba-kube-api-access-2q5nv\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443701 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443658 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-serving-certs-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443701 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443682 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443921 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443730 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-federate-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443921 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443753 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-metrics-client-ca\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.443921 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.443801 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.444592 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.444555 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.445244 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.445198 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-metrics-client-ca\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.445244 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.445235 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866403c0-bfc0-4e67-8424-1686b798b6ba-serving-certs-ca-bundle\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.446280 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.446260 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.447879 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.446905 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-secret-telemeter-client\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.447879 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.447086 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-telemeter-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.447879 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.447535 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/866403c0-bfc0-4e67-8424-1686b798b6ba-federate-client-tls\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.506402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.506377 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5nv\" (UniqueName: \"kubernetes.io/projected/866403c0-bfc0-4e67-8424-1686b798b6ba-kube-api-access-2q5nv\") pod \"telemeter-client-7f595c5795-zfd8j\" (UID: \"866403c0-bfc0-4e67-8424-1686b798b6ba\") " pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:40.600476 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:40.600448 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" Nov 02 22:49:41.021003 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.020952 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f595c5795-zfd8j"] Nov 02 22:49:41.023110 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:49:41.023066 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866403c0_bfc0_4e67_8424_1686b798b6ba.slice/crio-2f6503f9bcba93fe3e10586435e728ad34c12095c2fc03909e2429ea643556d6 WatchSource:0}: Error finding container 2f6503f9bcba93fe3e10586435e728ad34c12095c2fc03909e2429ea643556d6: Status 404 returned error can't find the container with id 2f6503f9bcba93fe3e10586435e728ad34c12095c2fc03909e2429ea643556d6 Nov 02 22:49:41.146001 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.145962 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"2fe075c35605721696c1bfc9692787ecfdbbf3ce3368f5884642f70676c0e30c"} Nov 02 22:49:41.146148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.146036 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"417cb5188806ca2747c7c776ac538fa134ab5e3b752972da660caa5ef7de391a"} Nov 02 22:49:41.146148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.146061 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"66912d2a91615540c404a98a3aa8a0f2f819d3036e86c26bde43394036d14390"} Nov 02 22:49:41.146148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.146085 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"7530b9b2b9c873a1dedbd01371476b6a6e3ecd6f0a0df465b47177ee299b4363"} Nov 02 22:49:41.146148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.146106 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"1ef7615cafc2f4f94abfaaa64ba0d1259e100e4030d0f560b19458fcb96e0c51"} Nov 02 22:49:41.146148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.146130 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26470e00-2773-4d56-b168-b3ce323c57df","Type":"ContainerStarted","Data":"5c765166a1c3ac72630da604848b19256a77cf17588d3afbf914270aac4c6a48"} Nov 02 22:49:41.147453 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.147430 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" event={"ID":"866403c0-bfc0-4e67-8424-1686b798b6ba","Type":"ContainerStarted","Data":"2f6503f9bcba93fe3e10586435e728ad34c12095c2fc03909e2429ea643556d6"} Nov 02 22:49:41.282149 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:41.282036 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.26912594 podStartE2EDuration="3.282023658s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.897718ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:41.280766818 +0000 UTC m=+95.710636680" watchObservedRunningTime="2026-11-02 22:49:41.282023658 +0000 UTC m=+95.711893512" Nov 02 22:49:42.253344 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253309 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:42.253770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253615 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="prometheus" containerID="cri-o://ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" gracePeriod=600 Nov 02 22:49:42.253770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253648 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy" containerID="cri-o://c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" gracePeriod=600 Nov 02 22:49:42.253770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253697 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-web" containerID="cri-o://4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" gracePeriod=600 Nov 02 22:49:42.253770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253677 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="config-reloader" containerID="cri-o://ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" gracePeriod=600 Nov 02 22:49:42.254026 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253660 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="thanos-sidecar" containerID="cri-o://99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" gracePeriod=600 Nov 02 22:49:42.254026 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.253655 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" gracePeriod=600 Nov 02 22:49:42.531960 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.531931 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666090 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666142 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666190 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666213 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666249 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666283 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666329 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666354 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666384 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666412 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666449 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666480 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666506 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666540 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666579 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666610 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666676 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666709 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666755 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.667103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.666785 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\") pod \"456d1d47-09a8-4217-98f0-613cf8494ea0\" (UID: \"456d1d47-09a8-4217-98f0-613cf8494ea0\") " Nov 02 22:49:42.668206 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.667216 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.668206 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.667945 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.669886 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.668905 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.669886 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.669680 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.671868 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.671823 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.672582 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.672219 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.672826 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.672798 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.673370 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.673344 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:42.674283 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.674256 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.674409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.674380 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:42.674409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.674391 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "kube-api-access-nvm8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:42.674883 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.674844 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:49:42.675549 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.675521 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.675732 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.675711 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.676485 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.676453 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.677693 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.677668 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.678330 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.678309 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.679002 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.678979 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.679270 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.679248 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:42.685950 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.685921 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config" pod "456d1d47-09a8-4217-98f0-613cf8494ea0" (UID: "456d1d47-09a8-4217-98f0-613cf8494ea0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:42.768325 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768290 4494 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-tls-assets\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768424 4494 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-kube-rbac-proxy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768451 4494 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768466 4494 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768481 4494 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768510 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768497 4494 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-metrics-client-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768513 4494 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768525 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvm8m\" (UniqueName: \"kubernetes.io/projected/456d1d47-09a8-4217-98f0-613cf8494ea0-kube-api-access-nvm8m\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768534 4494 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768543 4494 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768552 4494 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768560 4494 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-metrics-client-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768568 4494 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-config-out\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768576 4494 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768586 4494 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768596 4494 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-grpc-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768605 4494 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-web-config\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768613 4494 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/456d1d47-09a8-4217-98f0-613cf8494ea0-prometheus-k8s-db\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768622 4494 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456d1d47-09a8-4217-98f0-613cf8494ea0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:42.768765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:42.768630 4494 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/456d1d47-09a8-4217-98f0-613cf8494ea0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:49:43.159681 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159647 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" exitCode=0 Nov 02 22:49:43.159681 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159674 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" exitCode=0 Nov 02 22:49:43.159681 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159683 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" exitCode=0 Nov 02 22:49:43.159681 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159688 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" exitCode=0 Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159696 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" exitCode=0 Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159704 4494 generic.go:356] "Generic (PLEG): container finished" podID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" exitCode=0 Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159727 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159728 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159856 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159884 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159912 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159922 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159952 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} Nov 02 22:49:43.160007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.159990 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} Nov 02 22:49:43.160579 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.160028 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"456d1d47-09a8-4217-98f0-613cf8494ea0","Type":"ContainerDied","Data":"80a0ef00a8d03de7618305c9284e76132d5a1bda63dce7ed0b2ea2e4f5599572"} Nov 02 22:49:43.172552 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.172535 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.192151 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.192127 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.199502 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.199475 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.206494 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.206470 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.213927 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.213912 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.222558 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.222542 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.234756 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.234737 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.235033 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.235015 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.235105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235049 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.235105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235076 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.235375 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.235356 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.235418 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235388 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.235418 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235409 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.235762 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.235741 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.235838 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235770 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.235838 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.235790 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.236025 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.236009 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.236084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236033 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.236084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236053 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.236355 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.236336 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.236481 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236363 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.236481 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236381 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.236629 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.236613 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.236685 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236637 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.236685 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236656 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.236912 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:43.236890 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.236989 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236912 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.236989 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.236929 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.237200 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237160 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.237200 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237200 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.237495 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237474 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.237578 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237495 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.237775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237757 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.237825 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.237776 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.238047 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238027 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.238123 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238048 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.238338 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238320 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.238407 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238339 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.238521 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238505 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.238584 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238522 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.238742 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238727 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.238820 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238743 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.238969 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238949 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.239022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.238969 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.239203 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239162 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.239283 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239204 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.239462 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239438 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.239529 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239462 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.239691 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239669 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.239762 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239694 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.239933 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239915 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.239973 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.239933 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.240161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240144 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.240255 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240162 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.240437 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240418 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.240485 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240436 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.240669 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240650 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.240712 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240670 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.240885 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240870 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.240930 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.240885 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.241093 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241075 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.241126 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241092 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.241307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241290 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.241350 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241308 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.241495 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241476 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.241538 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241495 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.241686 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241672 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.241728 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241686 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.241888 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241871 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.241888 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.241887 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.242097 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242078 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.242186 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242108 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.242355 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242339 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.242417 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242356 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.242599 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242584 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.242664 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242600 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.242841 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242825 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.242879 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.242845 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.243076 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243059 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.243118 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243079 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.243345 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243288 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.243345 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243314 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.243531 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243514 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.243572 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243533 4494 scope.go:122] "RemoveContainer" containerID="f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022" Nov 02 22:49:43.243731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243714 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022"} err="failed to get container status \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": rpc error: code = NotFound desc = could not find container \"f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022\": container with ID starting with f749282abb441414bcdbb73fb31c031507bc8da260b95e866602b4dcfc2f8022 not found: ID does not exist" Nov 02 22:49:43.243768 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243731 4494 scope.go:122] "RemoveContainer" containerID="c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472" Nov 02 22:49:43.243936 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243920 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472"} err="failed to get container status \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": rpc error: code = NotFound desc = could not find container \"c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472\": container with ID starting with c87e8badf5d24850776f9e4811848e7058d08ace48b409263724ef7424bc2472 not found: ID does not exist" Nov 02 22:49:43.243977 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.243935 4494 scope.go:122] "RemoveContainer" containerID="4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3" Nov 02 22:49:43.244117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244102 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3"} err="failed to get container status \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": rpc error: code = NotFound desc = could not find container \"4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3\": container with ID starting with 4642c692734979eea127fbcdeb9b7b3468a7d21e99d6f76dedaf676854221cc3 not found: ID does not exist" Nov 02 22:49:43.244157 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244117 4494 scope.go:122] "RemoveContainer" containerID="99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11" Nov 02 22:49:43.244308 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244293 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11"} err="failed to get container status \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": rpc error: code = NotFound desc = could not find container \"99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11\": container with ID starting with 99d432d1b9ae76c4cbea9d86fb918d60f432d6a8e772eb362329fa0d42227f11 not found: ID does not exist" Nov 02 22:49:43.244350 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244308 4494 scope.go:122] "RemoveContainer" containerID="ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459" Nov 02 22:49:43.244483 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244469 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459"} err="failed to get container status \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": rpc error: code = NotFound desc = could not find container \"ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459\": container with ID starting with ba5fd9ac345339156267d2488d6e472275f4d0fa03b107b4189e434064fed459 not found: ID does not exist" Nov 02 22:49:43.244521 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244485 4494 scope.go:122] "RemoveContainer" containerID="ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d" Nov 02 22:49:43.244651 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244636 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d"} err="failed to get container status \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": rpc error: code = NotFound desc = could not find container \"ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d\": container with ID starting with ffe1d6bcd7c0226b8c48a9cd94e70baac84a094ce1ca0341e97233968536737d not found: ID does not exist" Nov 02 22:49:43.244651 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244650 4494 scope.go:122] "RemoveContainer" containerID="5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b" Nov 02 22:49:43.244821 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.244805 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b"} err="failed to get container status \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": rpc error: code = NotFound desc = could not find container \"5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b\": container with ID starting with 5a77eec40857ea149da833331e5d3b6eefcec23cbfa26ebb0716d90ded8fc07b not found: ID does not exist" Nov 02 22:49:43.330564 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.330540 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:43.381475 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.381456 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:43.382299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382280 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:43.382569 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382558 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="thanos-sidecar" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382571 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="thanos-sidecar" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382585 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382591 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382600 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-thanos" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382605 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-thanos" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382617 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="init-config-reloader" Nov 02 22:49:43.382620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382622 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="init-config-reloader" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382634 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-web" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382639 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-web" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382647 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="config-reloader" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382651 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="config-reloader" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382662 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="prometheus" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382667 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="prometheus" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382712 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382719 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-thanos" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382727 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="kube-rbac-proxy-web" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382736 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="config-reloader" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382744 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="prometheus" Nov 02 22:49:43.382829 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.382752 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" containerName="thanos-sidecar" Nov 02 22:49:43.390838 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.390816 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.444470 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.444396 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:49:43.444470 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.444407 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:49:43.444925 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.444901 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:49:43.445043 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.444940 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:49:43.445043 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.444981 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:49:43.445249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445227 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:49:43.445249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445243 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445301 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445301 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445305 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445328 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xcgqq\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445331 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8cslm08d5q9bp\"" Nov 02 22:49:43.445411 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445389 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:49:43.445667 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.445494 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:49:43.474082 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474061 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474213 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474106 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474213 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474135 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474303 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474212 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474303 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474249 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474303 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474290 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474429 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474318 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474429 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474393 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474488 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474440 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474488 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474474 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474494 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474509 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474551 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474610 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474600 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474661 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474757 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474783 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474934 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474805 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.474934 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.474834 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmnk\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-kube-api-access-vbmnk\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.496583 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.496558 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:49:43.496662 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.496602 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:49:43.502362 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.502337 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:43.575999 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.575968 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.575999 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576000 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576017 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576069 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576099 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576116 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576142 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576226 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576190 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576247 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576275 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576300 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576352 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576382 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576406 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576453 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.576491 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576477 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576499 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576524 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmnk\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-kube-api-access-vbmnk\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576593 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576713 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576886 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577117 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.576953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.577807 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.577783 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.579103 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.579076 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.579615 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.579590 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.580836 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.580808 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.580980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.580818 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.580980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.580859 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.580980 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.580910 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.581234 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.581215 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.581318 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.581300 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.581377 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.581360 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545d7ff-bdcb-40bd-981c-d6bfd5320434-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.581875 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.581845 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.581982 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.581958 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.582060 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.582043 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.582158 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.582142 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.582939 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.582917 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545d7ff-bdcb-40bd-981c-d6bfd5320434-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.583447 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.583431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545d7ff-bdcb-40bd-981c-d6bfd5320434-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.642356 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.642333 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmnk\" (UniqueName: \"kubernetes.io/projected/6545d7ff-bdcb-40bd-981c-d6bfd5320434-kube-api-access-vbmnk\") pod \"prometheus-k8s-0\" (UID: \"6545d7ff-bdcb-40bd-981c-d6bfd5320434\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:43.691946 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.691917 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456d1d47-09a8-4217-98f0-613cf8494ea0" path="/var/lib/kubelet/pods/456d1d47-09a8-4217-98f0-613cf8494ea0/volumes" Nov 02 22:49:43.714775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:43.714715 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:44.146053 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.146025 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:49:44.148297 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:49:44.148269 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6545d7ff_bdcb_40bd_981c_d6bfd5320434.slice/crio-eef0612b49163d496d065a7c85e026c69cdc297f229b0e74f1ecab90a2dc3f82 WatchSource:0}: Error finding container eef0612b49163d496d065a7c85e026c69cdc297f229b0e74f1ecab90a2dc3f82: Status 404 returned error can't find the container with id eef0612b49163d496d065a7c85e026c69cdc297f229b0e74f1ecab90a2dc3f82 Nov 02 22:49:44.167792 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.167758 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"eef0612b49163d496d065a7c85e026c69cdc297f229b0e74f1ecab90a2dc3f82"} Nov 02 22:49:44.170781 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.170754 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" event={"ID":"866403c0-bfc0-4e67-8424-1686b798b6ba","Type":"ContainerStarted","Data":"55ebbba500d2a08d533a4d3f4ecbe4fb7b2fe9040739c7eca17c488c4c9436ea"} Nov 02 22:49:44.170932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.170814 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" event={"ID":"866403c0-bfc0-4e67-8424-1686b798b6ba","Type":"ContainerStarted","Data":"73b53cefc9504f53af0d537a3682d61269b64d6f27f3bf22e85e924295bd0134"} Nov 02 22:49:44.170932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.170844 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" event={"ID":"866403c0-bfc0-4e67-8424-1686b798b6ba","Type":"ContainerStarted","Data":"127e714780ebe7a959950ece6747c42f3bb9620d352d77bd5670179c99291ec1"} Nov 02 22:49:44.299134 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.299079 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f595c5795-zfd8j" podStartSLOduration=2.122654794 podStartE2EDuration="4.299064357s" totalImagesPullingTime="2.176409563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:44.298447589 +0000 UTC m=+98.728317450" watchObservedRunningTime="2026-11-02 22:49:44.299064357 +0000 UTC m=+98.728934214" Nov 02 22:49:44.872209 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:44.872166 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:45.021121 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:45.021091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:45.177120 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:45.177090 4494 generic.go:356] "Generic (PLEG): container finished" podID="6545d7ff-bdcb-40bd-981c-d6bfd5320434" containerID="2a95ce038c5f70c73c4e3114935047f621f84d2a1c6fed774c31851f732a0ba2" exitCode=0 Nov 02 22:49:45.177281 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:45.177187 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerDied","Data":"2a95ce038c5f70c73c4e3114935047f621f84d2a1c6fed774c31851f732a0ba2"} Nov 02 22:49:45.396983 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:45.396950 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:45.550842 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:45.550670 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:46.184263 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184229 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"56676d5a3b8b26beba2de07540f2768b58baa338e0ba5f99e7b2988803f2ec73"} Nov 02 22:49:46.184682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184291 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"cb7cb41ce79156fa35874d910822101e0f892f1a880b1aed94921f169d0e4131"} Nov 02 22:49:46.184682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"247e747bbfcae04db880c651a880a86671071ad82955ec0f009a910a6259d147"} Nov 02 22:49:46.184682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184341 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"c0878534e5e6372715644651108ce84cbbe82a620d20f1ae0d2733d649b2d6ae"} Nov 02 22:49:46.184682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184369 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"fc7c3ea6de4cf5bb431d77b5ca9a477c15acba310f1f9f99caa58f2a73f8cff8"} Nov 02 22:49:46.184682 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.184396 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545d7ff-bdcb-40bd-981c-d6bfd5320434","Type":"ContainerStarted","Data":"a1005872242e3e1125fd2417f3a49d59658efd5f53cb74ea8edc66424a6d8e5b"} Nov 02 22:49:46.317791 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:46.317743 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.3010116910000002 podStartE2EDuration="3.317729257s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.717566ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:46.31674776 +0000 UTC m=+100.746617621" watchObservedRunningTime="2026-11-02 22:49:46.317729257 +0000 UTC m=+100.747599153" Nov 02 22:49:47.557458 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:49:47.557420 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:49:48.715538 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:48.715503 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:49:55.693710 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:49:55.693670 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:05.259423 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:05.259338 4494 generic.go:356] "Generic (PLEG): container finished" podID="c6318b9a-152d-4583-aad6-4c0878a0f111" containerID="2396b546cf1ca00b0d5677e9dd7b694ff53074ad6b4478d2678b734900ef00df" exitCode=0 Nov 02 22:50:05.259423 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:05.259409 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" event={"ID":"c6318b9a-152d-4583-aad6-4c0878a0f111","Type":"ContainerDied","Data":"2396b546cf1ca00b0d5677e9dd7b694ff53074ad6b4478d2678b734900ef00df"} Nov 02 22:50:06.405618 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.405590 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:50:06.478978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.478943 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\") pod \"c6318b9a-152d-4583-aad6-4c0878a0f111\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " Nov 02 22:50:06.479200 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479011 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\") pod \"c6318b9a-152d-4583-aad6-4c0878a0f111\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " Nov 02 22:50:06.479200 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479077 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\" (UniqueName: \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\") pod \"c6318b9a-152d-4583-aad6-4c0878a0f111\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " Nov 02 22:50:06.479200 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479123 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\") pod \"c6318b9a-152d-4583-aad6-4c0878a0f111\" (UID: \"c6318b9a-152d-4583-aad6-4c0878a0f111\") " Nov 02 22:50:06.479375 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479324 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path" pod "c6318b9a-152d-4583-aad6-4c0878a0f111" (UID: "c6318b9a-152d-4583-aad6-4c0878a0f111"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:06.479447 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479420 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle" pod "c6318b9a-152d-4583-aad6-4c0878a0f111" (UID: "c6318b9a-152d-4583-aad6-4c0878a0f111"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:06.479486 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479448 4494 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c6318b9a-152d-4583-aad6-4c0878a0f111-archives-path\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:06.479557 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.479540 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle" pod "c6318b9a-152d-4583-aad6-4c0878a0f111" (UID: "c6318b9a-152d-4583-aad6-4c0878a0f111"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:06.481370 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.481351 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b" pod "c6318b9a-152d-4583-aad6-4c0878a0f111" (UID: "c6318b9a-152d-4583-aad6-4c0878a0f111"). InnerVolumeSpecName "kube-api-access-bg48b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:06.580304 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.580212 4494 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-service-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:06.580304 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.580245 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg48b\" (UniqueName: \"kubernetes.io/projected/c6318b9a-152d-4583-aad6-4c0878a0f111-kube-api-access-bg48b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:06.580304 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:06.580256 4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6318b9a-152d-4583-aad6-4c0878a0f111-trusted-ca-bundle\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:50:07.268594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:07.268565 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" Nov 02 22:50:07.268786 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:07.268568 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kcxfk-jfxwl" event={"ID":"c6318b9a-152d-4583-aad6-4c0878a0f111","Type":"ContainerDied","Data":"e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2"} Nov 02 22:50:07.268786 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:07.268684 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a5c052c393393aa0fc997c5ad1cdaef2d808ffc22af811b66992c8d8da0ba2" Nov 02 22:50:08.357892 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:08.357855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:08.501648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:08.501613 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:10.432706 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:10.432672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:17.561647 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:50:17.561548 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:50:27.813121 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:27.813080 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:27.967279 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:27.967243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:37.572374 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:37.572312 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:38.195542 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:38.195488 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:43.715575 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:43.715541 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:50:43.732436 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:43.732410 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:50:44.423815 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:44.423790 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:50:47.557032 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:50:47.556984 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:50:52.160596 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:52.160555 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:52.306427 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:52.306394 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:56.039366 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:50:56.039327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:04.896911 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:04.896844 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:05.040796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:05.040764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:08.492563 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:08.492520 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:08.636598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:08.636565 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:08.878607 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:08.878517 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:09.022873 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:09.022836 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:14.873398 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:14.873369 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:15.023734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:15.023703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:17.556548 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:51:17.556508 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:51:47.556589 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:51:47.556506 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:51:56.390677 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:51:56.390642 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:17.556095 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:52:17.556051 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:52:35.238770 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:35.238726 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:47.556627 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:52:47.556580 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:52:55.785104 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:55.785072 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:56.004578 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:56.004533 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:56.744150 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:56.744106 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:57.316751 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:57.316710 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:57.466616 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:57.466584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:58.610693 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:52:58.610630 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:04.125712 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:04.125680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:04.287780 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:04.287750 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:05.579351 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:05.579315 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:05.896730 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:05.896697 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:06.041271 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:06.041233 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:09.658563 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:09.658478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:17.556572 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:53:17.556484 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:53:47.557787 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:53:47.557749 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:53:50.258964 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:50.258932 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:50.409779 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:50.409748 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:51.216893 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.216862 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:51.576686 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.576585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:51.754055 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.754025 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:51.843759 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.843679 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75"] Nov 02 22:53:51.844022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.844007 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c6318b9a-152d-4583-aad6-4c0878a0f111" containerName="insights-gathering" Nov 02 22:53:51.844022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.844020 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6318b9a-152d-4583-aad6-4c0878a0f111" containerName="insights-gathering" Nov 02 22:53:51.844097 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.844073 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6318b9a-152d-4583-aad6-4c0878a0f111" containerName="insights-gathering" Nov 02 22:53:51.847105 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.847090 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.901467 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.901439 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:53:51.902628 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.902594 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75"] Nov 02 22:53:51.936311 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936279 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.936498 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936318 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.936498 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936453 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.936628 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936513 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrsh\" (UniqueName: \"kubernetes.io/projected/8d684585-afe2-4c10-af6c-8a0b6fbc4752-kube-api-access-xcrsh\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.936628 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936547 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.936628 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.936573 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:51.950975 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.950934 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:53:51.951403 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.951382 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c5ts9\"" Nov 02 22:53:51.951499 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.951452 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:53:51.951499 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.951477 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:53:51.969309 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:51.969281 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:52.002227 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.002201 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:53:52.037258 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037227 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037258 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037259 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037291 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037414 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrsh\" (UniqueName: \"kubernetes.io/projected/8d684585-afe2-4c10-af6c-8a0b6fbc4752-kube-api-access-xcrsh\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037451 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037475 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.037723 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.037660 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-manifests\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.040140 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.040108 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-metrics-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.040375 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.040270 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.040436 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.040311 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d684585-afe2-4c10-af6c-8a0b6fbc4752-tmp\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.040563 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.040546 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d684585-afe2-4c10-af6c-8a0b6fbc4752-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.149118 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.149037 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrsh\" (UniqueName: \"kubernetes.io/projected/8d684585-afe2-4c10-af6c-8a0b6fbc4752-kube-api-access-xcrsh\") pod \"opendatahub-operator-controller-manager-79d467458d-vvg75\" (UID: \"8d684585-afe2-4c10-af6c-8a0b6fbc4752\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.161752 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.161720 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:52.221854 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.221795 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:52.432506 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.432476 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:52.665747 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.665724 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75"] Nov 02 22:53:52.666796 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:53:52.666770 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d684585_afe2_4c10_af6c_8a0b6fbc4752.slice/crio-0cd2abc26aeed04a7253af587c21e95f7039e8cb7d34d9abee78f12ea575cca7 WatchSource:0}: Error finding container 0cd2abc26aeed04a7253af587c21e95f7039e8cb7d34d9abee78f12ea575cca7: Status 404 returned error can't find the container with id 0cd2abc26aeed04a7253af587c21e95f7039e8cb7d34d9abee78f12ea575cca7 Nov 02 22:53:52.668370 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:52.668351 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:53:53.096220 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:53.096160 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" event={"ID":"8d684585-afe2-4c10-af6c-8a0b6fbc4752","Type":"ContainerStarted","Data":"0cd2abc26aeed04a7253af587c21e95f7039e8cb7d34d9abee78f12ea575cca7"} Nov 02 22:53:53.863896 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:53.863865 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:54.068867 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:54.068829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:57.092594 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:57.092535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:57.114490 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:57.114458 4494 generic.go:356] "Generic (PLEG): container finished" podID="8d684585-afe2-4c10-af6c-8a0b6fbc4752" containerID="989705df9fbb56221ce71fb0da6180721f5b71cfd33c5406d39562586dcf45c0" exitCode=0 Nov 02 22:53:57.114647 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:57.114547 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" event={"ID":"8d684585-afe2-4c10-af6c-8a0b6fbc4752","Type":"ContainerDied","Data":"989705df9fbb56221ce71fb0da6180721f5b71cfd33c5406d39562586dcf45c0"} Nov 02 22:53:57.838527 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:57.838492 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:58.004013 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:58.003988 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:58.121228 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:58.121117 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" event={"ID":"8d684585-afe2-4c10-af6c-8a0b6fbc4752","Type":"ContainerStarted","Data":"e4b1612e5a39014396ad2dc8159138705d0c186d62a40b579a7e3c4845e75082"} Nov 02 22:53:58.121676 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:58.121226 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:53:58.173333 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:58.173301 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:58.268755 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:53:58.268693 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" podStartSLOduration=2.559973183 podStartE2EDuration="7.268677634s" totalImagesPullingTime="4.214608289s" totalInitContainerRuntime="494.096162ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:58.266562033 +0000 UTC m=+352.696431893" watchObservedRunningTime="2026-11-02 22:53:58.268677634 +0000 UTC m=+352.698547494" Nov 02 22:54:00.284275 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:00.284241 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:02.647529 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:02.647484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:02.816053 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:02.816011 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:04.200710 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:04.200670 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:04.380500 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:04.380463 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:05.455113 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.455077 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz"] Nov 02 22:54:05.462434 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.462413 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.518376 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.518353 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz"] Nov 02 22:54:05.518921 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.518905 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:54:05.558358 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.558329 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lmn\" (UniqueName: \"kubernetes.io/projected/8119a0d9-4070-4c25-8f76-c0321329e40d-kube-api-access-j6lmn\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.558513 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.558365 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8119a0d9-4070-4c25-8f76-c0321329e40d-tmp\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.567641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.567621 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7rlxp\"" Nov 02 22:54:05.567641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.567621 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:54:05.658873 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.658843 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lmn\" (UniqueName: \"kubernetes.io/projected/8119a0d9-4070-4c25-8f76-c0321329e40d-kube-api-access-j6lmn\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.659023 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.658881 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8119a0d9-4070-4c25-8f76-c0321329e40d-tmp\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.659219 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.659203 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8119a0d9-4070-4c25-8f76-c0321329e40d-tmp\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:05.777472 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.777383 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:54:05.841357 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.841325 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:54:05.848731 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:05.848704 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lmn\" (UniqueName: \"kubernetes.io/projected/8119a0d9-4070-4c25-8f76-c0321329e40d-kube-api-access-j6lmn\") pod \"jobset-operator-674cd59dc5-8ggvz\" (UID: \"8119a0d9-4070-4c25-8f76-c0321329e40d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:06.137095 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:06.137017 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7rlxp\"" Nov 02 22:54:06.145068 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:06.145050 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" Nov 02 22:54:06.520755 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:06.520728 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz"] Nov 02 22:54:06.523147 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:06.523108 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8119a0d9_4070_4c25_8f76_c0321329e40d.slice/crio-cd9c54c6fba0eb17c6404debeba464c5503b3b4c970c645844e713088508b895 WatchSource:0}: Error finding container cd9c54c6fba0eb17c6404debeba464c5503b3b4c970c645844e713088508b895: Status 404 returned error can't find the container with id cd9c54c6fba0eb17c6404debeba464c5503b3b4c970c645844e713088508b895 Nov 02 22:54:07.158245 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:07.158206 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" event={"ID":"8119a0d9-4070-4c25-8f76-c0321329e40d","Type":"ContainerStarted","Data":"cd9c54c6fba0eb17c6404debeba464c5503b3b4c970c645844e713088508b895"} Nov 02 22:54:09.127798 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:09.127771 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79d467458d-vvg75" Nov 02 22:54:12.661000 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:12.660965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:12.888624 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:12.888580 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:16.704916 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:16.704880 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:17.001994 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:17.001858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:17.560910 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:54:17.560822 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:54:19.034760 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:19.034714 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:19.364343 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:19.364261 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:19.691247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:19.691216 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:20.210465 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:20.210430 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" event={"ID":"8119a0d9-4070-4c25-8f76-c0321329e40d","Type":"ContainerStarted","Data":"64f03013b9de7fc93711ee78bec8b06967de1c01d09adbe4fbadf3bd2cd1da8a"} Nov 02 22:54:20.340098 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:20.340043 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8ggvz" podStartSLOduration=2.670593207 podStartE2EDuration="15.340028283s" totalImagesPullingTime="12.669435076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:20.339355829 +0000 UTC m=+374.769225693" watchObservedRunningTime="2026-11-02 22:54:20.340028283 +0000 UTC m=+374.769898144" Nov 02 22:54:20.385393 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:20.385362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:21.003606 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:21.003573 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:21.166316 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:21.166280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:23.927765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:23.927725 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q"] Nov 02 22:54:23.931134 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:23.931118 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.026853 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.026818 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ff8\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-kube-api-access-26ff8\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.026853 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.026861 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.042010 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.041983 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nnbh2\"" Nov 02 22:54:24.042010 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.041998 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:54:24.042220 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.042035 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:54:24.043567 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.043551 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q"] Nov 02 22:54:24.128225 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.128166 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ff8\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-kube-api-access-26ff8\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.128384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.128235 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.159942 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.159911 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:24.200188 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.200108 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.200422 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.200400 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ff8\" (UniqueName: \"kubernetes.io/projected/2c98c574-1006-4578-907a-3b1f4d289716-kube-api-access-26ff8\") pod \"cert-manager-cainjector-5c57bbd4fc-tqn5q\" (UID: \"2c98c574-1006-4578-907a-3b1f4d289716\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.242527 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.242503 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" Nov 02 22:54:24.515410 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.515331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:24.617456 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:24.617401 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q"] Nov 02 22:54:24.619734 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:24.619699 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c98c574_1006_4578_907a_3b1f4d289716.slice/crio-cc0ae6dd30be7d44c337e6c659bf7db8232ef7c0ee540c57809f5a6dfd48fc6c WatchSource:0}: Error finding container cc0ae6dd30be7d44c337e6c659bf7db8232ef7c0ee540c57809f5a6dfd48fc6c: Status 404 returned error can't find the container with id cc0ae6dd30be7d44c337e6c659bf7db8232ef7c0ee540c57809f5a6dfd48fc6c Nov 02 22:54:25.231689 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:25.231645 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" event={"ID":"2c98c574-1006-4578-907a-3b1f4d289716","Type":"ContainerStarted","Data":"cc0ae6dd30be7d44c337e6c659bf7db8232ef7c0ee540c57809f5a6dfd48fc6c"} Nov 02 22:54:26.164785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:26.164749 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:26.439761 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:26.439675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:26.596571 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:26.596541 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:27.169330 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:27.169295 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:27.484878 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:27.484794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:27.707994 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:27.707957 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:30.617794 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:30.617767 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:31.643089 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:31.643057 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:33.570926 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:33.570896 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:33.720813 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:33.720783 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:33.966088 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:33.966052 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:34.110353 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:34.110313 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:37.420479 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.418606 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-twcvh"] Nov 02 22:54:37.420479 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.419506 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:37.422857 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.422831 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.476668 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.476636 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twcvh"] Nov 02 22:54:37.525827 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.525784 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-88pbb\"" Nov 02 22:54:37.544278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.544251 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-bound-sa-token\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.544424 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.544407 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6csm\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-kube-api-access-k6csm\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.597517 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.597479 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:37.645190 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.645142 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6csm\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-kube-api-access-k6csm\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.645375 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.645240 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-bound-sa-token\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.712048 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.711950 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-bound-sa-token\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.712817 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.712793 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6csm\" (UniqueName: \"kubernetes.io/projected/7f9978d8-c162-4f2b-a42d-9716a2df7490-kube-api-access-k6csm\") pod \"cert-manager-597b95cb84-twcvh\" (UID: \"7f9978d8-c162-4f2b-a42d-9716a2df7490\") " pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.736689 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.736664 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twcvh" Nov 02 22:54:37.768974 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:37.768943 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:38.085148 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.085061 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:38.121022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.120996 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twcvh"] Nov 02 22:54:38.123187 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:38.123145 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9978d8_c162_4f2b_a42d_9716a2df7490.slice/crio-794719fa029461f466de672398ebb6a01c30f40457e9bc616dcf3e7a4caaf5b4 WatchSource:0}: Error finding container 794719fa029461f466de672398ebb6a01c30f40457e9bc616dcf3e7a4caaf5b4: Status 404 returned error can't find the container with id 794719fa029461f466de672398ebb6a01c30f40457e9bc616dcf3e7a4caaf5b4 Nov 02 22:54:38.287582 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.287550 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" event={"ID":"2c98c574-1006-4578-907a-3b1f4d289716","Type":"ContainerStarted","Data":"d6161e05f588ff672d9a44ab9f092bc81b8a7a29f04759077555f8bab6d8fae8"} Nov 02 22:54:38.289096 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.289071 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twcvh" event={"ID":"7f9978d8-c162-4f2b-a42d-9716a2df7490","Type":"ContainerStarted","Data":"0b12b5fa9bc56365c2326a8f84a61487ef27878c212675fdc1b9d327e5b28b1a"} Nov 02 22:54:38.289260 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.289106 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twcvh" event={"ID":"7f9978d8-c162-4f2b-a42d-9716a2df7490","Type":"ContainerStarted","Data":"794719fa029461f466de672398ebb6a01c30f40457e9bc616dcf3e7a4caaf5b4"} Nov 02 22:54:38.407418 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.407309 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tqn5q" podStartSLOduration=2.183621064 podStartE2EDuration="15.407296573s" totalImagesPullingTime="13.223675509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:38.406586832 +0000 UTC m=+392.836456694" watchObservedRunningTime="2026-11-02 22:54:38.407296573 +0000 UTC m=+392.837166434" Nov 02 22:54:38.529452 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:38.529398 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-twcvh" podStartSLOduration=1.5293822609999999 podStartE2EDuration="1.529382261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:38.527838998 +0000 UTC m=+392.957708858" watchObservedRunningTime="2026-11-02 22:54:38.529382261 +0000 UTC m=+392.959252121" Nov 02 22:54:39.547866 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:39.547826 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:39.774317 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:39.774283 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:40.006059 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.006028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:40.026466 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.026434 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc"] Nov 02 22:54:40.029895 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.029876 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.066350 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.066323 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.066486 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.066388 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tmp\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.066486 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.066471 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r28j7\" (UniqueName: \"kubernetes.io/projected/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-kube-api-access-r28j7\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.084844 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.084817 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:54:40.084996 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.084953 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:54:40.136252 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.136228 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc"] Nov 02 22:54:40.166920 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.166889 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tmp\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.167063 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.166959 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r28j7\" (UniqueName: \"kubernetes.io/projected/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-kube-api-access-r28j7\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.167063 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.166995 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.167707 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.167675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:40.169368 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.169345 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tmp\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.170952 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.170936 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:54:40.171063 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.171046 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 22:54:40.171108 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.171058 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jpp79\"" Nov 02 22:54:40.180311 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.180292 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-tls-certs\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.229823 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.229790 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r28j7\" (UniqueName: \"kubernetes.io/projected/edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca-kube-api-access-r28j7\") pod \"kube-auth-proxy-65dbc8865f-r6rbc\" (UID: \"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca\") " pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.342060 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.341990 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" Nov 02 22:54:40.724584 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.724559 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc"] Nov 02 22:54:40.725601 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:40.725574 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb2e3a6_f7f3_4333_b74e_7ebeceaf70ca.slice/crio-e740d61f307800444256c08cadaf58a04e857ea46a7cc81c2d222eb37039c58e WatchSource:0}: Error finding container e740d61f307800444256c08cadaf58a04e857ea46a7cc81c2d222eb37039c58e: Status 404 returned error can't find the container with id e740d61f307800444256c08cadaf58a04e857ea46a7cc81c2d222eb37039c58e Nov 02 22:54:40.823989 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:40.823960 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:41.051275 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:41.051194 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:41.302679 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:41.302586 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" event={"ID":"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca","Type":"ContainerStarted","Data":"e740d61f307800444256c08cadaf58a04e857ea46a7cc81c2d222eb37039c58e"} Nov 02 22:54:41.517000 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:41.516968 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:41.726367 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:41.726337 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:42.225688 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.225614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:42.325985 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.325940 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:54:42.329986 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.329955 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.393034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.392339 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:54:42.393034 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.392989 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 22:54:42.405415 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.405384 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.405558 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.405420 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsdt\" (UniqueName: \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.442497 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.442469 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:54:42.442990 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.442970 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v4tz9\"" Nov 02 22:54:42.444359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.444328 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:54:42.506759 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.506690 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.506759 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.506734 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsdt\" (UniqueName: \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.507611 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.507584 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.564899 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.564866 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:42.572778 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.572751 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsdt\" (UniqueName: \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\") pod \"limitador-limitador-74dfcfb894-fktjh\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.594252 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.592925 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:54:42.645935 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.645902 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:42.734530 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.731327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:42.882979 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:42.882859 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:43.032997 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:43.032968 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:54:43.034972 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:43.034941 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6312df5d_e5b3_4071_b998_8c3e5e27dcda.slice/crio-5fb13002bce1ee3b8990ae3281530ec2c33fc82119b03ee23b92eda4f5eed106 WatchSource:0}: Error finding container 5fb13002bce1ee3b8990ae3281530ec2c33fc82119b03ee23b92eda4f5eed106: Status 404 returned error can't find the container with id 5fb13002bce1ee3b8990ae3281530ec2c33fc82119b03ee23b92eda4f5eed106 Nov 02 22:54:43.046748 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:43.046725 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:43.313964 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:43.313923 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" event={"ID":"6312df5d-e5b3-4071-b998-8c3e5e27dcda","Type":"ContainerStarted","Data":"5fb13002bce1ee3b8990ae3281530ec2c33fc82119b03ee23b92eda4f5eed106"} Nov 02 22:54:44.015748 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:44.015716 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:44.163249 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:44.163216 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:44.320086 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:44.320006 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" event={"ID":"edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca","Type":"ContainerStarted","Data":"72be1001ee22b9624321830fa78d66b988c163f7a9da287418e7585975e67a94"} Nov 02 22:54:44.440392 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:44.440345 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65dbc8865f-r6rbc" podStartSLOduration=2.561804071 podStartE2EDuration="5.440330849s" totalImagesPullingTime="2.878526778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:44.43864581 +0000 UTC m=+398.868515670" watchObservedRunningTime="2026-11-02 22:54:44.440330849 +0000 UTC m=+398.870200707" Nov 02 22:54:46.704964 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.704937 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:54:46.708296 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.708280 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:54:46.718442 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.718419 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:54:46.721536 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.721521 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:46.770419 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.770395 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dpz6v\"" Nov 02 22:54:46.772111 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.772088 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:54:46.778480 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.778456 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:54:46.848377 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.848296 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:46.848533 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.848437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2n6\" (UniqueName: \"kubernetes.io/projected/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-kube-api-access-2v2n6\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:46.848533 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.848463 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-config-file\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:46.848533 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.848499 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zpz\" (UniqueName: \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\") pod \"authorino-56c6c9ddd5-cqgrl\" (UID: \"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e\") " pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:54:46.950012 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.949975 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zpz\" (UniqueName: \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\") pod \"authorino-56c6c9ddd5-cqgrl\" (UID: \"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e\") " pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:54:46.950204 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.950103 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2n6\" (UniqueName: \"kubernetes.io/projected/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-kube-api-access-2v2n6\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:46.950204 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.950134 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-config-file\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:46.950882 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:46.950846 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-config-file\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:47.011689 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.011656 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zpz\" (UniqueName: \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\") pod \"authorino-56c6c9ddd5-cqgrl\" (UID: \"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e\") " pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:54:47.011855 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.011839 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2n6\" (UniqueName: \"kubernetes.io/projected/5d0a16e7-cd0c-4161-80ea-cdb46ae08b91-kube-api-access-2v2n6\") pod \"limitador-limitador-7c9b87d574-f8dkp\" (UID: \"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:47.018422 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.018398 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:54:47.020084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.020067 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:54:47.033155 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.033136 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:47.118519 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.118455 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:47.336014 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.335984 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" event={"ID":"6312df5d-e5b3-4071-b998-8c3e5e27dcda","Type":"ContainerStarted","Data":"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9"} Nov 02 22:54:47.336193 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.336122 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:54:47.390981 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.390898 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:54:47.433130 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:47.433101 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5deec6b6_e322_4810_9a6b_3fd31d9c6b3e.slice/crio-a2848c427ed2cff83de05045908de2e9aa6c45aad66a0ee273672a8aa035b70b WatchSource:0}: Error finding container a2848c427ed2cff83de05045908de2e9aa6c45aad66a0ee273672a8aa035b70b: Status 404 returned error can't find the container with id a2848c427ed2cff83de05045908de2e9aa6c45aad66a0ee273672a8aa035b70b Nov 02 22:54:47.448396 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.448372 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:54:47.450088 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:54:47.450065 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0a16e7_cd0c_4161_80ea_cdb46ae08b91.slice/crio-48dc2fb8888655e6fc52658ec274f7c24877587fe6fcdc011f076ade9303b7ad WatchSource:0}: Error finding container 48dc2fb8888655e6fc52658ec274f7c24877587fe6fcdc011f076ade9303b7ad: Status 404 returned error can't find the container with id 48dc2fb8888655e6fc52658ec274f7c24877587fe6fcdc011f076ade9303b7ad Nov 02 22:54:47.455982 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.455882 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" podStartSLOduration=1.9011871500000002 podStartE2EDuration="5.455869174s" totalImagesPullingTime="3.554682024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:47.455792786 +0000 UTC m=+401.885662647" watchObservedRunningTime="2026-11-02 22:54:47.455869174 +0000 UTC m=+401.885739073" Nov 02 22:54:47.559206 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:54:47.559099 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:54:47.741399 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:47.741363 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:48.098352 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.098272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:48.266982 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.266952 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:48.341938 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.341906 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" event={"ID":"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e","Type":"ContainerStarted","Data":"a2848c427ed2cff83de05045908de2e9aa6c45aad66a0ee273672a8aa035b70b"} Nov 02 22:54:48.343461 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.343436 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" event={"ID":"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91","Type":"ContainerStarted","Data":"0da19c4c05451c7bb45b039a48def45bcc3b361873f3a6ea516998dc6f759b01"} Nov 02 22:54:48.343579 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.343483 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" event={"ID":"5d0a16e7-cd0c-4161-80ea-cdb46ae08b91","Type":"ContainerStarted","Data":"48dc2fb8888655e6fc52658ec274f7c24877587fe6fcdc011f076ade9303b7ad"} Nov 02 22:54:48.477384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:48.477337 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" podStartSLOduration=2.477320975 podStartE2EDuration="2.477320975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:48.475661456 +0000 UTC m=+402.905531319" watchObservedRunningTime="2026-11-02 22:54:48.477320975 +0000 UTC m=+402.907190836" Nov 02 22:54:49.348719 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:49.348688 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:54:51.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:51.359152 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" event={"ID":"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e","Type":"ContainerStarted","Data":"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16"} Nov 02 22:54:51.475604 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:51.475546 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" podStartSLOduration=2.500229621 podStartE2EDuration="5.475531051s" totalImagesPullingTime="2.97530143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:51.474586791 +0000 UTC m=+405.904456652" watchObservedRunningTime="2026-11-02 22:54:51.475531051 +0000 UTC m=+405.905400931" Nov 02 22:54:52.710002 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:52.709963 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:52.856000 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:52.855960 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:56.320985 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:56.320939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:56.474834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:56.474782 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:57.420163 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:57.420128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:58.344701 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:54:58.344649 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:55:00.354565 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:00.354541 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8dkp" Nov 02 22:55:00.524250 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:00.524158 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:55:00.524405 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:00.524350 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" podUID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" containerName="limitador" containerID="cri-o://59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9" gracePeriod=30 Nov 02 22:55:01.059559 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.059535 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:55:01.076692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.076667 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\" (UniqueName: \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\") pod \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " Nov 02 22:55:01.076812 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.076702 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\" (UniqueName: \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\") pod \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\" (UID: \"6312df5d-e5b3-4071-b998-8c3e5e27dcda\") " Nov 02 22:55:01.077101 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.077080 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file" pod "6312df5d-e5b3-4071-b998-8c3e5e27dcda" (UID: "6312df5d-e5b3-4071-b998-8c3e5e27dcda"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:55:01.078936 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.078911 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt" pod "6312df5d-e5b3-4071-b998-8c3e5e27dcda" (UID: "6312df5d-e5b3-4071-b998-8c3e5e27dcda"). InnerVolumeSpecName "kube-api-access-wpsdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:55:01.177561 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.177528 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsdt\" (UniqueName: \"kubernetes.io/projected/6312df5d-e5b3-4071-b998-8c3e5e27dcda-kube-api-access-wpsdt\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:55:01.177561 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.177559 4494 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6312df5d-e5b3-4071-b998-8c3e5e27dcda-config-file\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:55:01.402683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.402600 4494 generic.go:356] "Generic (PLEG): container finished" podID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" containerID="59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9" exitCode=0 Nov 02 22:55:01.402683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.402653 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" Nov 02 22:55:01.402683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.402677 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" event={"ID":"6312df5d-e5b3-4071-b998-8c3e5e27dcda","Type":"ContainerDied","Data":"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9"} Nov 02 22:55:01.403197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.402727 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fktjh" event={"ID":"6312df5d-e5b3-4071-b998-8c3e5e27dcda","Type":"ContainerDied","Data":"5fb13002bce1ee3b8990ae3281530ec2c33fc82119b03ee23b92eda4f5eed106"} Nov 02 22:55:01.403197 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.402758 4494 scope.go:122] "RemoveContainer" containerID="59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9" Nov 02 22:55:01.411725 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.411701 4494 scope.go:122] "RemoveContainer" containerID="59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9" Nov 02 22:55:01.411992 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:55:01.411971 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9\": container with ID starting with 59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9 not found: ID does not exist" containerID="59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9" Nov 02 22:55:01.412041 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.412002 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9"} err="failed to get container status \"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9\": rpc error: code = NotFound desc = could not find container \"59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9\": container with ID starting with 59d3d9ffe847ae96d89ae9757b083db50b2cf9ac8228e31ea16c104a9201c4e9 not found: ID does not exist" Nov 02 22:55:01.569004 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.568972 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:55:01.619953 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.619918 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fktjh"] Nov 02 22:55:01.691284 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.691254 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" path="/var/lib/kubelet/pods/6312df5d-e5b3-4071-b998-8c3e5e27dcda/volumes" Nov 02 22:55:01.719834 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.719803 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:01.867670 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:01.867598 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:02.015493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:02.015409 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:02.167605 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:02.167568 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:05.102487 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.102447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:05.246640 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.246609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:05.764196 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.764153 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px"] Nov 02 22:55:05.764531 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.764519 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" containerName="limitador" Nov 02 22:55:05.764575 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.764532 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" containerName="limitador" Nov 02 22:55:05.764616 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.764607 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6312df5d-e5b3-4071-b998-8c3e5e27dcda" containerName="limitador" Nov 02 22:55:05.768787 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.768772 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.819541 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.819512 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-data\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.819726 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.819641 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22gp\" (UniqueName: \"kubernetes.io/projected/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-kube-api-access-d22gp\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.824269 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.824248 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 22:55:05.825083 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.825064 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px"] Nov 02 22:55:05.843134 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.843117 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:55:05.860305 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.860277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:05.874051 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.874032 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:55:05.874192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.874079 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wdnvq\"" Nov 02 22:55:05.920928 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.920890 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22gp\" (UniqueName: \"kubernetes.io/projected/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-kube-api-access-d22gp\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.921066 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.920951 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-data\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.921327 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.921306 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-data\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:05.987774 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:05.987697 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22gp\" (UniqueName: \"kubernetes.io/projected/8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1-kube-api-access-d22gp\") pod \"postgres-7cdbf6dc44-g98px\" (UID: \"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:06.080477 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:06.080434 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:06.452443 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:06.452422 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px"] Nov 02 22:55:06.454130 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:55:06.454102 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3ff83b_6adb_4a10_bc0b_7a7f1247a2e1.slice/crio-2eb132973668119cb94831e3d4bb809ba32f750278763b607cd23ba4c1f2afca WatchSource:0}: Error finding container 2eb132973668119cb94831e3d4bb809ba32f750278763b607cd23ba4c1f2afca: Status 404 returned error can't find the container with id 2eb132973668119cb94831e3d4bb809ba32f750278763b607cd23ba4c1f2afca Nov 02 22:55:07.251102 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:07.251071 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:07.422262 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:07.422228 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:07.429575 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:07.429548 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" event={"ID":"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1","Type":"ContainerStarted","Data":"2eb132973668119cb94831e3d4bb809ba32f750278763b607cd23ba4c1f2afca"} Nov 02 22:55:14.466391 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:14.466355 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" event={"ID":"8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1","Type":"ContainerStarted","Data":"7a54f456fadfa6480eb71d3c7cbd2c2c21479686d082d1092b8145375adceb09"} Nov 02 22:55:14.466708 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:14.466490 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:14.584584 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:14.584531 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" podStartSLOduration=1.6534232530000001 podStartE2EDuration="9.584515498s" totalImagesPullingTime="7.931092245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:14.583109695 +0000 UTC m=+429.012979557" watchObservedRunningTime="2026-11-02 22:55:14.584515498 +0000 UTC m=+429.014385359" Nov 02 22:55:17.558012 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:55:17.557973 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:55:20.501876 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:20.501842 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g98px" Nov 02 22:55:27.490020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.489984 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:27.551020 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.550989 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:27.556635 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.556615 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.609984 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.609956 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.610135 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.610017 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpkx\" (UniqueName: \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.610135 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.610100 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.612656 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.612634 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:55:27.613242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.612899 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:55:27.613242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.613043 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:55:27.613242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.613095 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:55:27.613242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.613132 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:55:27.613242 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.613217 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p68w6\"" Nov 02 22:55:27.613459 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.613409 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:27.637043 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.637019 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:27.710493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.710474 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.710635 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.710511 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpkx\" (UniqueName: \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.710635 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.710531 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.713072 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.713050 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.713165 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.713086 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.771969 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.771905 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpkx\" (UniqueName: \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\") pod \"maas-controller-59fd54f5bc-nj7zt\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.782348 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.782323 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:27.870922 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.870886 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:27.931224 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:27.931190 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:28.130259 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.130193 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 02 22:55:28.134308 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.134294 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.198868 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.196029 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 02 22:55:28.213828 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.213802 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.213987 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.213858 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rll2n\" (UniqueName: \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.213987 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.213914 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.243585 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.243562 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:28.245743 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:55:28.245717 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8465bc_5b07_45da_9dc5_cdbce6b80bdd.slice/crio-77e47cc3c835603c58bf38237f9915dcb705367e428d6a88dd589f587a008c24 WatchSource:0}: Error finding container 77e47cc3c835603c58bf38237f9915dcb705367e428d6a88dd589f587a008c24: Status 404 returned error can't find the container with id 77e47cc3c835603c58bf38237f9915dcb705367e428d6a88dd589f587a008c24 Nov 02 22:55:28.314451 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.314419 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.314648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.314480 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rll2n\" (UniqueName: \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.314648 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.314500 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.317062 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.317037 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.317199 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.317095 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.376215 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.376187 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rll2n\" (UniqueName: \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\") pod \"maas-controller-7457749d7d-nzlvl\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.447958 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.447930 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:28.536275 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.536140 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" event={"ID":"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd","Type":"ContainerStarted","Data":"77e47cc3c835603c58bf38237f9915dcb705367e428d6a88dd589f587a008c24"} Nov 02 22:55:28.820381 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:28.820349 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 02 22:55:28.822571 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:55:28.822532 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98647bf_90df_4e01_8959_b857f423b272.slice/crio-4cbf02ffb159e0deccf1e373ea4f5ca576e7aeb135cd574541e1a3b9c2eea00a WatchSource:0}: Error finding container 4cbf02ffb159e0deccf1e373ea4f5ca576e7aeb135cd574541e1a3b9c2eea00a: Status 404 returned error can't find the container with id 4cbf02ffb159e0deccf1e373ea4f5ca576e7aeb135cd574541e1a3b9c2eea00a Nov 02 22:55:29.541579 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:29.541541 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerStarted","Data":"4cbf02ffb159e0deccf1e373ea4f5ca576e7aeb135cd574541e1a3b9c2eea00a"} Nov 02 22:55:32.193402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.193370 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:32.354575 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.354497 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:32.558764 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.558728 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" event={"ID":"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd","Type":"ContainerStarted","Data":"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6"} Nov 02 22:55:32.558992 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.558820 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:32.560381 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.560361 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerStarted","Data":"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b"} Nov 02 22:55:32.560488 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.560479 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:32.678861 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.678792 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" podStartSLOduration=2.179794746 podStartE2EDuration="5.67877814s" totalImagesPullingTime="3.498983394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:32.677163266 +0000 UTC m=+447.107033127" watchObservedRunningTime="2026-11-02 22:55:32.67877814 +0000 UTC m=+447.108647995" Nov 02 22:55:32.797438 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:32.797384 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7457749d7d-nzlvl" podStartSLOduration=1.6780762280000001 podStartE2EDuration="4.797369933s" totalImagesPullingTime="3.119293705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:32.796238645 +0000 UTC m=+447.226108505" watchObservedRunningTime="2026-11-02 22:55:32.797369933 +0000 UTC m=+447.227239794" Nov 02 22:55:35.316873 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:35.316846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:35.876641 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:35.876597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:36.916449 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:36.916409 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:37.061006 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:37.060967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:43.566733 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:43.566699 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:43.567247 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:43.566784 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:55:43.739918 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:43.739885 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:43.740159 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:43.740135 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" podUID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" containerName="manager" containerID="cri-o://b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6" gracePeriod=30 Nov 02 22:55:43.987664 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:43.987643 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:44.162144 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.162119 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\" (UniqueName: \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\") pod \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " Nov 02 22:55:44.162302 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.162196 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\") pod \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " Nov 02 22:55:44.162389 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.162366 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\") pod \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\" (UID: \"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd\") " Nov 02 22:55:44.164535 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.164504 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx" pod "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" (UID: "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd"). InnerVolumeSpecName "kube-api-access-scpkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:55:44.164677 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.164582 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs" pod "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" (UID: "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:55:44.164677 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.164594 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert" pod "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" (UID: "ce8465bc-5b07-45da-9dc5-cdbce6b80bdd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:55:44.263243 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.263210 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scpkx\" (UniqueName: \"kubernetes.io/projected/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-kube-api-access-scpkx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:55:44.263243 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.263238 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:55:44.263243 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.263248 4494 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd-webhook-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:55:44.615692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.615599 4494 generic.go:356] "Generic (PLEG): container finished" podID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" containerID="b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6" exitCode=0 Nov 02 22:55:44.615692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.615645 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" event={"ID":"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd","Type":"ContainerDied","Data":"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6"} Nov 02 22:55:44.615692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.615648 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" Nov 02 22:55:44.615692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.615691 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-nj7zt" event={"ID":"ce8465bc-5b07-45da-9dc5-cdbce6b80bdd","Type":"ContainerDied","Data":"77e47cc3c835603c58bf38237f9915dcb705367e428d6a88dd589f587a008c24"} Nov 02 22:55:44.616357 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.615716 4494 scope.go:122] "RemoveContainer" containerID="b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6" Nov 02 22:55:44.625078 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.625057 4494 scope.go:122] "RemoveContainer" containerID="b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6" Nov 02 22:55:44.625506 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:55:44.625487 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6\": container with ID starting with b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6 not found: ID does not exist" containerID="b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6" Nov 02 22:55:44.625589 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.625513 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6"} err="failed to get container status \"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6\": rpc error: code = NotFound desc = could not find container \"b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6\": container with ID starting with b1145e9753d8f0e8c7faa63bcafc0bdbeda591819b166130b80d36c58b2723e6 not found: ID does not exist" Nov 02 22:55:44.782751 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.782715 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:44.784788 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:44.784768 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-nj7zt"] Nov 02 22:55:45.692349 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:45.692311 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" path="/var/lib/kubelet/pods/ce8465bc-5b07-45da-9dc5-cdbce6b80bdd/volumes" Nov 02 22:55:47.560355 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:55:47.560320 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:55:54.589128 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.589089 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m"] Nov 02 22:55:54.589665 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.589528 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" containerName="manager" Nov 02 22:55:54.589665 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.589541 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" containerName="manager" Nov 02 22:55:54.589665 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.589618 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8465bc-5b07-45da-9dc5-cdbce6b80bdd" containerName="manager" Nov 02 22:55:54.593047 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.593031 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.646278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.646236 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:55:54.646278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.646236 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:55:54.646278 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.646270 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:55:54.646585 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.646289 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:55:54.646585 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.646457 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqkr\"" Nov 02 22:55:54.698786 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.698758 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m"] Nov 02 22:55:54.764362 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764327 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764362 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764365 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/039a073e-0b49-4ab9-b6f6-79ece1993578-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764645 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764386 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwlv\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-kube-api-access-4zwlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764645 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764477 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764645 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764530 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764645 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764558 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.764645 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.764635 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865502 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865570 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/039a073e-0b49-4ab9-b6f6-79ece1993578-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865603 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865602 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwlv\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-kube-api-access-4zwlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865641 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865674 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865698 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.865884 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.865754 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.866404 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.866379 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.868186 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.868151 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/039a073e-0b49-4ab9-b6f6-79ece1993578-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.868442 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.868416 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.868557 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.868540 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.868694 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.868675 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.927809 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.927770 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:54.927989 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:54.927968 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwlv\" (UniqueName: \"kubernetes.io/projected/039a073e-0b49-4ab9-b6f6-79ece1993578-kube-api-access-4zwlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2645m\" (UID: \"039a073e-0b49-4ab9-b6f6-79ece1993578\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:55.205066 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:55.205030 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:55.632239 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:55.632210 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m"] Nov 02 22:55:55.633961 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:55:55.633926 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039a073e_0b49_4ab9_b6f6_79ece1993578.slice/crio-6a2654968097b137fed247a40aef1d305f50a3cee7a6d53b327d1904d4350018 WatchSource:0}: Error finding container 6a2654968097b137fed247a40aef1d305f50a3cee7a6d53b327d1904d4350018: Status 404 returned error can't find the container with id 6a2654968097b137fed247a40aef1d305f50a3cee7a6d53b327d1904d4350018 Nov 02 22:55:55.665734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:55.665704 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" event={"ID":"039a073e-0b49-4ab9-b6f6-79ece1993578","Type":"ContainerStarted","Data":"6a2654968097b137fed247a40aef1d305f50a3cee7a6d53b327d1904d4350018"} Nov 02 22:55:56.499452 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:56.499420 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:57.767493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:57.767456 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:59.450189 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.450142 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:59.541729 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.541699 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:55:59.541822 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.541760 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:55:59.595192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.595148 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:59.696537 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.696496 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" event={"ID":"039a073e-0b49-4ab9-b6f6-79ece1993578","Type":"ContainerStarted","Data":"f5b43ff5d96bdf1f1b3b70d9c7eac0a1810c3e5c0efadce0c3d1fc87f199a870"} Nov 02 22:55:59.696690 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.696632 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:55:59.742340 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.742269 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:59.829875 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.829822 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" podStartSLOduration=1.9249242180000001 podStartE2EDuration="5.829808715s" totalImagesPullingTime="3.904884497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:59.828574176 +0000 UTC m=+474.258444043" watchObservedRunningTime="2026-11-02 22:55:59.829808715 +0000 UTC m=+474.259678570" Nov 02 22:55:59.887088 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:55:59.887059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:00.030318 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.030230 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:00.174937 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.174905 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:00.359263 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.359190 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:00.510348 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.510314 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:00.704024 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.703990 4494 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2645m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:56:00.704217 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:00.704047 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" podUID="039a073e-0b49-4ab9-b6f6-79ece1993578" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:56:01.671638 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:01.671605 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:02.505692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:02.505653 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:02.655699 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:02.655668 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:03.703043 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:03.703011 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2645m" Nov 02 22:56:04.651881 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:04.651845 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:04.816978 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:04.816947 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:04.962053 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:04.962020 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:05.105634 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:05.105606 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:06.807643 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:06.807608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:06.953260 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:06.953219 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.101338 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.101236 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.248157 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.248122 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.397638 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.397562 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.541327 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.541292 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.685693 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.685645 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.833802 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.833772 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:07.977626 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:07.977544 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:11.278504 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:11.278466 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:56:17.560215 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:56:17.560111 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:56:19.714882 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:19.714843 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:20.076344 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:20.076229 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:20.256636 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:20.256586 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:23.276621 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:23.276583 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:23.421524 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:23.421487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:23.884118 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:23.884089 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:56:23.884329 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:23.884309 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" podUID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" containerName="authorino" containerID="cri-o://53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16" gracePeriod=30 Nov 02 22:56:24.137651 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.137576 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:56:24.240602 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.240570 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\" (UniqueName: \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\") pod \"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e\" (UID: \"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e\") " Nov 02 22:56:24.242932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.242905 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz" pod "5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" (UID: "5deec6b6-e322-4810-9a6b-3fd31d9c6b3e"). InnerVolumeSpecName "kube-api-access-m2zpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:24.342060 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.342025 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zpz\" (UniqueName: \"kubernetes.io/projected/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e-kube-api-access-m2zpz\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:56:24.808251 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.808218 4494 generic.go:356] "Generic (PLEG): container finished" podID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" containerID="53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16" exitCode=0 Nov 02 22:56:24.808455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.808269 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" Nov 02 22:56:24.808455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.808301 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" event={"ID":"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e","Type":"ContainerDied","Data":"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16"} Nov 02 22:56:24.808455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.808343 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cqgrl" event={"ID":"5deec6b6-e322-4810-9a6b-3fd31d9c6b3e","Type":"ContainerDied","Data":"a2848c427ed2cff83de05045908de2e9aa6c45aad66a0ee273672a8aa035b70b"} Nov 02 22:56:24.808455 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.808360 4494 scope.go:122] "RemoveContainer" containerID="53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16" Nov 02 22:56:24.817395 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.817379 4494 scope.go:122] "RemoveContainer" containerID="53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16" Nov 02 22:56:24.817644 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:56:24.817625 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16\": container with ID starting with 53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16 not found: ID does not exist" containerID="53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16" Nov 02 22:56:24.817694 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.817653 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16"} err="failed to get container status \"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16\": rpc error: code = NotFound desc = could not find container \"53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16\": container with ID starting with 53413cbeca7465b022ac3c8d5e161b81e4a6ce07c0fb42da6e92556039cfef16 not found: ID does not exist" Nov 02 22:56:24.972771 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.972736 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:56:24.978849 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:24.978819 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cqgrl"] Nov 02 22:56:25.698272 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:25.698234 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" path="/var/lib/kubelet/pods/5deec6b6-e322-4810-9a6b-3fd31d9c6b3e/volumes" Nov 02 22:56:47.592956 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:56:47.592876 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:56:56.316578 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:56.316531 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:58.114188 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:56:58.114118 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:12.914609 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:12.914575 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:13.063567 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:13.063533 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:13.681629 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:13.681596 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:13.826744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:13.826709 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:17.563075 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:57:17.563037 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:57:18.906146 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:18.906104 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:29.295676 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:29.295628 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:39.725166 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:39.725129 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:47.557697 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:57:47.557606 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:57:58.465598 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:57:58.465565 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:01.242766 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:01.242729 4494 generic.go:356] "Generic (PLEG): container finished" podID="d98647bf-90df-4e01-8959-b857f423b272" containerID="48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b" exitCode=1 Nov 02 22:58:01.243192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:01.242800 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerDied","Data":"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b"} Nov 02 22:58:01.243192 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:01.243131 4494 scope.go:122] "RemoveContainer" containerID="48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b" Nov 02 22:58:02.147685 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:02.147632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:02.250671 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:02.250636 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerStarted","Data":"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a"} Nov 02 22:58:02.251069 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:02.250979 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:58:02.497493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:02.497466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:10.269290 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:10.269257 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:10.416265 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:10.416225 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:13.257079 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:13.257051 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 02 22:58:17.561133 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:58:17.561089 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:58:32.821875 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:32.821832 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:33.101946 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:33.101856 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:33.609582 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:33.609547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:34.852800 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:34.852770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:34.999883 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:34.999849 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:41.182042 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:41.182007 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:44.927735 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:44.927702 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:58:47.578420 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:58:47.578378 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:58:48.982493 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:48.982461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:52.382787 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:52.382756 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:58:53.879452 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.879422 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8"] Nov 02 22:58:53.882683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.880431 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" containerName="authorino" Nov 02 22:58:53.882683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.880470 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" containerName="authorino" Nov 02 22:58:53.882683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.880710 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5deec6b6-e322-4810-9a6b-3fd31d9c6b3e" containerName="authorino" Nov 02 22:58:53.887614 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.887591 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:53.941210 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.941156 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tzw2x\"" Nov 02 22:58:53.943556 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.943525 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8"] Nov 02 22:58:53.976566 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.976539 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:53.991866 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.991842 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 22:58:53.991963 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.991891 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:58:53.991963 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:53.991937 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:58:54.040271 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040237 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.040439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040279 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.040439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040298 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.040439 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040349 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd2l\" (UniqueName: \"kubernetes.io/projected/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-kube-api-access-mfd2l\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.040543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.040543 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.040523 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141081 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.140997 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141081 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141032 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141081 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141049 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd2l\" (UniqueName: \"kubernetes.io/projected/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-kube-api-access-mfd2l\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141081 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141074 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141442 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141442 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141247 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141551 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141512 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141590 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141572 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.141683 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.141659 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.143499 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.143477 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.143766 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.143746 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.203625 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.203589 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd2l\" (UniqueName: \"kubernetes.io/projected/39d7a7ef-1d53-4972-8b3f-8a2d8e499481-kube-api-access-mfd2l\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wkpk8\" (UID: \"39d7a7ef-1d53-4972-8b3f-8a2d8e499481\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.500932 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.500903 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:54.882402 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.882380 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8"] Nov 02 22:58:54.884407 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:58:54.884384 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d7a7ef_1d53_4972_8b3f_8a2d8e499481.slice/crio-badbd6b50d535fd102a673bb2177272603335e0cbe47857d29907e7cbeaa7285 WatchSource:0}: Error finding container badbd6b50d535fd102a673bb2177272603335e0cbe47857d29907e7cbeaa7285: Status 404 returned error can't find the container with id badbd6b50d535fd102a673bb2177272603335e0cbe47857d29907e7cbeaa7285 Nov 02 22:58:54.885765 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:54.885747 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:58:55.488404 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:55.488370 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" event={"ID":"39d7a7ef-1d53-4972-8b3f-8a2d8e499481","Type":"ContainerStarted","Data":"badbd6b50d535fd102a673bb2177272603335e0cbe47857d29907e7cbeaa7285"} Nov 02 22:58:57.499409 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:57.499372 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" event={"ID":"39d7a7ef-1d53-4972-8b3f-8a2d8e499481","Type":"ContainerStarted","Data":"c60bcd37c55eba5702ac60e71f701c761014d80c95f8196df92830dba5c2fb46"} Nov 02 22:58:57.501674 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:57.501656 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:58:57.621678 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:57.621628 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" podStartSLOduration=2.930798178 podStartE2EDuration="4.621614002s" totalImagesPullingTime="1.690815824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:57.62023666 +0000 UTC m=+652.050106522" watchObservedRunningTime="2026-11-02 22:58:57.621614002 +0000 UTC m=+652.051483863" Nov 02 22:58:58.303116 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.303079 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz"] Nov 02 22:58:58.308436 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.308357 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.362997 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.362971 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 22:58:58.388604 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.388573 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:58.415297 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.415264 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz"] Nov 02 22:58:58.479714 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479685 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.479877 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479717 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lm9r\" (UniqueName: \"kubernetes.io/projected/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-kube-api-access-7lm9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.479877 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479737 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.479877 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479808 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.479877 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479860 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.480029 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.479937 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.580983 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.580905 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.580983 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.580938 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lm9r\" (UniqueName: \"kubernetes.io/projected/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-kube-api-access-7lm9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.580983 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.580963 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.580986 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.581011 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.581032 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.581463 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.581530 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.581597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.581541 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.583620 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.583599 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.583744 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.583723 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.643110 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.643079 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lm9r\" (UniqueName: \"kubernetes.io/projected/1a6e11e2-f926-4ad3-8a2d-9cf289ededc1-kube-api-access-7lm9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bxwpz\" (UID: \"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:58.814582 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.814521 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:58.923859 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:58.923817 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:58:59.295279 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:59.295255 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz"] Nov 02 22:58:59.296952 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:58:59.296926 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6e11e2_f926_4ad3_8a2d_9cf289ededc1.slice/crio-4a32750ef5f82cfbcbedda01fcf831b96c9a1852b5300284738136d9c70c757d WatchSource:0}: Error finding container 4a32750ef5f82cfbcbedda01fcf831b96c9a1852b5300284738136d9c70c757d: Status 404 returned error can't find the container with id 4a32750ef5f82cfbcbedda01fcf831b96c9a1852b5300284738136d9c70c757d Nov 02 22:58:59.508962 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:58:59.508928 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" event={"ID":"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1","Type":"ContainerStarted","Data":"4a32750ef5f82cfbcbedda01fcf831b96c9a1852b5300284738136d9c70c757d"} Nov 02 22:59:00.515145 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:00.515116 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" event={"ID":"1a6e11e2-f926-4ad3-8a2d-9cf289ededc1","Type":"ContainerStarted","Data":"fa4c3799bc5ce4a7287d02bcc17a92a88c5781aa293eaecbe50b25c3f88e1bb6"} Nov 02 22:59:00.635481 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:00.635427 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" podStartSLOduration=1.948028282 podStartE2EDuration="2.635413805s" totalImagesPullingTime="687.385523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:00.633822855 +0000 UTC m=+655.063692717" watchObservedRunningTime="2026-11-02 22:59:00.635413805 +0000 UTC m=+655.065283665" Nov 02 22:59:00.924065 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:00.924033 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:59:01.326613 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.326531 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr"] Nov 02 22:59:01.330235 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.330219 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.405861 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.405834 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 22:59:01.421330 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.421304 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:01.435869 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.435844 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr"] Nov 02 22:59:01.508426 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508394 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.508591 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9q56\" (UniqueName: \"kubernetes.io/projected/475f4005-c975-46da-838c-c455dd86f04d-kube-api-access-w9q56\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.508591 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508534 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.508591 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508574 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.508692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475f4005-c975-46da-838c-c455dd86f04d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.508692 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.508658 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.609868 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609784 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.609868 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609832 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9q56\" (UniqueName: \"kubernetes.io/projected/475f4005-c975-46da-838c-c455dd86f04d-kube-api-access-w9q56\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.609868 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609867 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609899 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609934 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475f4005-c975-46da-838c-c455dd86f04d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.609969 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.610329 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610384 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.610358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.610556 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.610406 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.612431 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.612401 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475f4005-c975-46da-838c-c455dd86f04d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.612619 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.612600 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475f4005-c975-46da-838c-c455dd86f04d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.671992 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.671959 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9q56\" (UniqueName: \"kubernetes.io/projected/475f4005-c975-46da-838c-c455dd86f04d-kube-api-access-w9q56\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr\" (UID: \"475f4005-c975-46da-838c-c455dd86f04d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:01.943332 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:01.943288 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:02.318057 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:02.318033 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr"] Nov 02 22:59:02.320164 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:59:02.320142 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475f4005_c975_46da_838c_c455dd86f04d.slice/crio-09104868639fafec456254bb9708c4db5a3caf78af5597242be5c42347dc159e WatchSource:0}: Error finding container 09104868639fafec456254bb9708c4db5a3caf78af5597242be5c42347dc159e: Status 404 returned error can't find the container with id 09104868639fafec456254bb9708c4db5a3caf78af5597242be5c42347dc159e Nov 02 22:59:02.525775 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:02.525741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" event={"ID":"475f4005-c975-46da-838c-c455dd86f04d","Type":"ContainerStarted","Data":"09104868639fafec456254bb9708c4db5a3caf78af5597242be5c42347dc159e"} Nov 02 22:59:03.531882 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:03.531844 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" event={"ID":"475f4005-c975-46da-838c-c455dd86f04d","Type":"ContainerStarted","Data":"e70cb02d59bd3d90365b6707133c473d0efd32712afbbe37572aa3c09e9ce8aa"} Nov 02 22:59:03.660251 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:03.658607 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" podStartSLOduration=1.987642525 podStartE2EDuration="2.65858665s" totalImagesPullingTime="670.944125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:03.657368595 +0000 UTC m=+658.087238456" watchObservedRunningTime="2026-11-02 22:59:03.65858665 +0000 UTC m=+658.088456512" Nov 02 22:59:03.695629 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:03.695601 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:59:03.944220 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:03.944188 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:04.501194 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:04.501148 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:59:04.514338 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:04.514314 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:59:04.547991 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:04.547963 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wkpk8" Nov 02 22:59:06.521993 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:06.521945 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:08.310259 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.310228 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 22:59:08.314084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.314064 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367268 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367241 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367414 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367281 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367414 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367315 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367505 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367425 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 02 22:59:08.367505 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367425 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367505 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367488 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367608 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367514 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367608 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367532 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367688 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367621 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.367688 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.367639 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxp9m\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.419890 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.419858 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 22:59:08.468364 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468332 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468534 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468367 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468534 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468419 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468534 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468444 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxp9m\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468534 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468491 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468538 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468580 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468637 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.468785 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.469005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468872 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.469005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468946 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.469005 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.468972 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.469161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.469063 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.469382 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.469358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.470982 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.470959 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.471078 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.470980 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.530606 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.530569 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.530754 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.530616 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxp9m\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\") pod \"maas-default-gateway-openshift-default-694db55c87-6gqk7\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.626349 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.626263 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:08.924848 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.924810 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:59:08.937348 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:08.937319 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:59:09.050390 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:09.050369 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 22:59:09.051857 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:59:09.051835 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c2d72c_58a3_44cb_b0a9_12a9c3ac493c.slice/crio-672cc8448eaa2fd8010f1cddd5b8590c171904bebe85b85dc4861cded28bafe0 WatchSource:0}: Error finding container 672cc8448eaa2fd8010f1cddd5b8590c171904bebe85b85dc4861cded28bafe0: Status 404 returned error can't find the container with id 672cc8448eaa2fd8010f1cddd5b8590c171904bebe85b85dc4861cded28bafe0 Nov 02 22:59:09.561891 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:09.561853 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" event={"ID":"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c","Type":"ContainerStarted","Data":"672cc8448eaa2fd8010f1cddd5b8590c171904bebe85b85dc4861cded28bafe0"} Nov 02 22:59:09.573128 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:09.573106 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bxwpz" Nov 02 22:59:10.614801 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:10.614768 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:11.943707 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:11.943647 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:11.963359 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:11.963329 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:12.507882 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.507853 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:12.507965 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.507918 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:12.507965 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.507956 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:12.578788 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.578756 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" event={"ID":"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c","Type":"ContainerStarted","Data":"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff"} Nov 02 22:59:12.590446 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.590421 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr" Nov 02 22:59:12.626686 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.626659 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:12.701438 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:12.701390 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podStartSLOduration=1.24822606 podStartE2EDuration="4.701376576s" totalImagesPullingTime="3.453150516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:12.700890215 +0000 UTC m=+667.130760077" watchObservedRunningTime="2026-11-02 22:59:12.701376576 +0000 UTC m=+667.131246437" Nov 02 22:59:13.628472 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:13.628436 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:13.628862 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:13.628481 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:14.363898 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:14.363858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:14.626886 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:14.626814 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:14.626886 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:14.626870 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:15.627204 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:15.627147 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:15.627690 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:15.627249 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:16.627466 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:16.627432 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:16.627842 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:16.627494 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:17.588360 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:59:17.588273 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:59:17.627026 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:17.626998 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:17.627216 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:17.627046 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:17.676296 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:17.676262 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 22:59:18.627283 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:18.627248 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:18.627465 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:18.627303 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:19.627597 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:19.627566 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:59:19.628007 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:19.627617 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 02 22:59:21.668337 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:21.668303 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:59:21.668804 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:21.668358 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:59:22.269193 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:22.269136 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:22.270666 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:22.269536 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:22.270961 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:22.270943 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 22:59:23.440784 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.440745 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 22:59:23.444084 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.444068 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527572 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527542 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527572 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527779 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527619 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527779 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527729 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527843 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527787 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527843 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527818 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrz4\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527906 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527842 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527939 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527910 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.527973 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.527936 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.543116 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.543089 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 22:59:23.629035 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629004 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629035 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629037 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629099 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629158 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629212 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629252 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrz4\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629299 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629289 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629580 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629360 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629580 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629409 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629580 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629544 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629580 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629560 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629670 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.629796 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629761 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.630015 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.629987 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.631921 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.631900 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.632327 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.632298 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.691489 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.691424 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.691489 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.691467 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrz4\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\") pod \"maas-default-gateway-openshift-default-694db55c87-bxqdn\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:23.756601 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:23.756571 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:24.180806 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.180783 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 22:59:24.182564 ip-10-0-135-179 kubenswrapper[4494]: W1102 22:59:24.182537 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a4b29d_5679_456f_9aef_4e2a9082336f.slice/crio-84897d1235ebb62185951d098d787bda01a7ba983ca3a27eb230132eb328bb63 WatchSource:0}: Error finding container 84897d1235ebb62185951d098d787bda01a7ba983ca3a27eb230132eb328bb63: Status 404 returned error can't find the container with id 84897d1235ebb62185951d098d787bda01a7ba983ca3a27eb230132eb328bb63 Nov 02 22:59:24.185734 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.185706 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:24.185811 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.185766 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:24.185811 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.185794 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 22:59:24.640398 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.640318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" event={"ID":"23a4b29d-5679-456f-9aef-4e2a9082336f","Type":"ContainerStarted","Data":"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06"} Nov 02 22:59:24.640398 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.640395 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" event={"ID":"23a4b29d-5679-456f-9aef-4e2a9082336f","Type":"ContainerStarted","Data":"84897d1235ebb62185951d098d787bda01a7ba983ca3a27eb230132eb328bb63"} Nov 02 22:59:24.757124 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.757084 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:24.763674 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:24.763623 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podStartSLOduration=1.763608932 podStartE2EDuration="1.763608932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:24.762218391 +0000 UTC m=+679.192088253" watchObservedRunningTime="2026-11-02 22:59:24.763608932 +0000 UTC m=+679.193478797" Nov 02 22:59:25.759307 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:25.759272 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:25.759818 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:25.759331 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:26.757698 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:26.757653 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:26.757910 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:26.757713 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:27.757022 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:27.756980 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:27.757443 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:27.757050 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:28.757328 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:28.757287 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:28.757733 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:28.757345 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:29.757837 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:29.757792 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:29.758318 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:29.757906 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:30.756979 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:30.756937 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:59:30.757160 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:30.757006 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 22:59:32.441161 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:32.441123 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:32.441635 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:32.441472 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:32.442376 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:32.442342 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 22:59:41.465815 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:41.465776 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:41.618653 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:41.618618 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:42.981966 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:42.981924 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:43.140808 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:43.140763 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:47.559632 ip-10-0-135-179 kubenswrapper[4494]: E1102 22:59:47.559590 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 22:59:59.162049 ip-10-0-135-179 kubenswrapper[4494]: I1102 22:59:59.161997 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:11.872043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:11.872003 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.035147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.035110 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.191198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.191127 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.375057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.375007 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.523800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.523708 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:00:12.527251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.527231 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.585076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.585049 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f779f4ae-openshift-default-dockercfg-62wc8\"" Nov 02 23:00:12.609215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609153 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5cj\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609216 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609304 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609370 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609449 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609488 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609571 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609603 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.609657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.609632 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.655457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.655431 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:00:12.681328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.681293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.710545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710514 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5cj\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710546 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710563 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710617 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710651 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710680 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710720 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.710769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710747 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.710888 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.711082 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.711104 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.711290 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.711400 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.711631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.711610 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.713329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.713306 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.713488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.713472 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.773468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.773437 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5cj\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.773945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.773902 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\") pod \"e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.836441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.836413 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.839309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.839294 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:12.866711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.866681 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:00:12.870900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.870878 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.913049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.912997 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914315 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914392 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914419 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnln\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914453 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914531 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914563 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914595 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.914870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.914628 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:12.931773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.931420 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:00:12.940227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.937071 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7d7bf49f-openshift-default-dockercfg-psl8c\"" Nov 02 23:00:12.980939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:12.980912 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.015201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015143 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015217 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015246 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnln\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015276 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015340 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015364 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015411 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015455 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015559 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015595 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015660 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015887 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.015926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.015908 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.016461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.016435 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.017731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.017711 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.018043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.018024 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.078524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.078451 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.078688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.078650 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnln\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\") pod \"e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.188767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.188724 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:13.224784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.224752 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.283333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.283300 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:00:13.283461 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:00:13.283363 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be0d79e_06f2_40ce_98d2_e8514a0c8895.slice/crio-3fb964de45a0df2bf3e0ba85358217b852585e39bb7fb89ab240a5fc03f7d9e4 WatchSource:0}: Error finding container 3fb964de45a0df2bf3e0ba85358217b852585e39bb7fb89ab240a5fc03f7d9e4: Status 404 returned error can't find the container with id 3fb964de45a0df2bf3e0ba85358217b852585e39bb7fb89ab240a5fc03f7d9e4 Nov 02 23:00:13.285791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.285754 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.285883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.285833 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.285939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.285922 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.485837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.485803 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:00:13.489775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.489755 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520546 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520588 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520618 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520643 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520741 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520771 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520818 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.520938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.520842 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622xp\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.544484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.544457 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fed9e8ee-gw-openshift-default-dockercfg-6c9fv\"" Nov 02 23:00:13.556618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.556584 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:00:13.576830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.576805 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:00:13.584514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.584477 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.584634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.584560 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.584634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.584602 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:13.588757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.588735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.621818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621791 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.621935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621831 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.621935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621851 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.621935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621870 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.621935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621898 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.621999 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622023 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622066 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622090 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622xp\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622334 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622646 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622662 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.622880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.622839 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.626493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.626470 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.626716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.626668 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.684338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.684298 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.684489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.684350 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-622xp\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\") pod \"e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.736799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.736764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.802659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.802623 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:13.884191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.884141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.904006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.903974 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerStarted","Data":"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3"} Nov 02 23:00:13.904209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.904027 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerStarted","Data":"3fb964de45a0df2bf3e0ba85358217b852585e39bb7fb89ab240a5fc03f7d9e4"} Nov 02 23:00:13.905685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.905663 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerStarted","Data":"88afd1adfae64cfdc78c9a648295f86110879dac050989f98b4f21a512f6843f"} Nov 02 23:00:13.905790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:13.905699 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerStarted","Data":"72d2c78c73b6bafcf0f3594bcdcb4afb76b1a5549f9735bd790be2427bbb4e96"} Nov 02 23:00:14.027409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.027346 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podStartSLOduration=2.02732644 podStartE2EDuration="2.02732644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:14.026339247 +0000 UTC m=+728.456209111" watchObservedRunningTime="2026-11-02 23:00:14.02732644 +0000 UTC m=+728.457196302" Nov 02 23:00:14.161284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.161233 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podStartSLOduration=2.161218479 podStartE2EDuration="2.161218479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:14.1602927 +0000 UTC m=+728.590162568" watchObservedRunningTime="2026-11-02 23:00:14.161218479 +0000 UTC m=+728.591088340" Nov 02 23:00:14.184329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.184293 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:00:14.194204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.189533 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:14.194204 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:00:14.189980 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c115ebb_fcf9_4bfb_8667_f72f4a933fa8.slice/crio-5249bdf3590977ac02c331c7a4c7c2babbe07df486d2a97852b191156dae21e2 WatchSource:0}: Error finding container 5249bdf3590977ac02c331c7a4c7c2babbe07df486d2a97852b191156dae21e2: Status 404 returned error can't find the container with id 5249bdf3590977ac02c331c7a4c7c2babbe07df486d2a97852b191156dae21e2 Nov 02 23:00:14.194204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.193129 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:14.194204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.193234 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:14.194204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.193279 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:14.210466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.210407 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:14.840322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.840289 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:14.845579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.845557 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:14.912236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.912205 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerStarted","Data":"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655"} Nov 02 23:00:14.912420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.912271 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:14.912420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.912319 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:14.912420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.912343 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerStarted","Data":"5249bdf3590977ac02c331c7a4c7c2babbe07df486d2a97852b191156dae21e2"} Nov 02 23:00:14.913396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.913375 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:14.913529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:14.913440 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:00:15.096233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.096134 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:15.097417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.097365 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podStartSLOduration=2.09734784 podStartE2EDuration="2.09734784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:15.095230815 +0000 UTC m=+729.525100675" watchObservedRunningTime="2026-11-02 23:00:15.09734784 +0000 UTC m=+729.527217702" Nov 02 23:00:15.411043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.411015 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:15.803546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.803458 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:15.809769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.809743 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:15.921544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.921508 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:15.924921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:15.924895 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:16.262228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:16.262161 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:16.426692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:16.426660 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:16.579210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:16.579110 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:16.866227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:16.866109 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:17.133741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:17.133641 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:17.301408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:17.301369 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:17.559711 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:00:17.559670 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:00:17.900209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:17.900097 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:18.108001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:18.107971 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:18.964739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:18.964692 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:19.130834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:19.130799 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:19.276080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:19.276003 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:19.890635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:19.890587 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:20.186882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:20.186845 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:20.423640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:20.423600 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:20.600907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:20.600815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:22.297006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:22.296964 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:22.656295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:22.656088 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:22.832132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:22.832095 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:22.996001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:22.995967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:23.169547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:23.169507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:25.290887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:25.290852 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:25.470034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:25.470002 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:26.575210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:26.575149 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:26.746920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:26.746886 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:26.919125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:26.919091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:27.772571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:27.772535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:27.954456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:27.954417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:28.234495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:28.234462 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:28.513931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:28.513849 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:28.661935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:28.661902 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:30.452461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:30.452428 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:30.619226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:30.619162 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:31.013023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.012991 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:00:31.016770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.016747 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.074901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.074872 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:00:31.122664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.122636 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5dd28258\"" Nov 02 23:00:31.122826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.122723 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5dd28258-dockercfg-p2drx\"" Nov 02 23:00:31.136412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.136377 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pvr\" (UniqueName: \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.136570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.136549 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.237415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.237382 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.237592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.237468 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pvr\" (UniqueName: \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.238105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.238088 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.300292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.300218 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pvr\" (UniqueName: \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\") pod \"payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.330918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.330888 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:31.704535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:31.704508 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:00:31.705992 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:00:31.705966 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b05de54_35e1_44b0_b5b4_ca49e4df3c15.slice/crio-21bfac74deda43ed0964a8dc3b1731cba718c2ff971e9af053ec16e9a279a1e0 WatchSource:0}: Error finding container 21bfac74deda43ed0964a8dc3b1731cba718c2ff971e9af053ec16e9a279a1e0: Status 404 returned error can't find the container with id 21bfac74deda43ed0964a8dc3b1731cba718c2ff971e9af053ec16e9a279a1e0 Nov 02 23:00:32.024328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:32.024226 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" event={"ID":"1b05de54-35e1-44b0-b5b4-ca49e4df3c15","Type":"ContainerStarted","Data":"21bfac74deda43ed0964a8dc3b1731cba718c2ff971e9af053ec16e9a279a1e0"} Nov 02 23:00:33.769582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:33.769543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:34.030438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:34.030338 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:34.178468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:34.178425 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:34.304009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:34.303944 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:00:35.062849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:35.062811 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" event={"ID":"1b05de54-35e1-44b0-b5b4-ca49e4df3c15","Type":"ContainerStarted","Data":"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071"} Nov 02 23:00:35.063282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:35.062920 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:35.183314 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:35.183260 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" podStartSLOduration=2.356602449 podStartE2EDuration="5.183246185s" totalImagesPullingTime="2.826643736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:35.181356363 +0000 UTC m=+749.611226224" watchObservedRunningTime="2026-11-02 23:00:35.183246185 +0000 UTC m=+749.613116047" Nov 02 23:00:38.990076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:38.990045 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:39.806016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:39.805986 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:39.992542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:39.992499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:40.223299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.223264 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:00:40.226700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.226685 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.283827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.283797 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-708f46a0\"" Nov 02 23:00:40.345964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.345920 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-708f46a0-dockercfg-xqz9b\"" Nov 02 23:00:40.346373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.346348 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:00:40.365641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.365608 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgx7\" (UniqueName: \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.365807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.365753 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.403655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.403603 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:00:40.407861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.407834 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.437428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.437392 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:00:40.466946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.466913 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.467140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.467092 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgx7\" (UniqueName: \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.467686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.467663 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.531574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.531545 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgx7\" (UniqueName: \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\") pod \"payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.559775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.559743 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:40.569729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.569687 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.569877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.569856 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfxh\" (UniqueName: \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.671793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.671753 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfxh\" (UniqueName: \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.671976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.671859 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.672687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.672660 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.733846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.733813 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfxh\" (UniqueName: \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\") pod \"payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:40.947603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:40.947571 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:00:40.949315 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:00:40.949288 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e6a984_3db4_4e54_93da_f857384f5316.slice/crio-13c23cf68448238080c5ee02779af852902448bb7368669c0b5eff597aba6b0f WatchSource:0}: Error finding container 13c23cf68448238080c5ee02779af852902448bb7368669c0b5eff597aba6b0f: Status 404 returned error can't find the container with id 13c23cf68448238080c5ee02779af852902448bb7368669c0b5eff597aba6b0f Nov 02 23:00:41.027282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.027255 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:41.097639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.097601 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" event={"ID":"c5e6a984-3db4-4e54-93da-f857384f5316","Type":"ContainerStarted","Data":"e971df1cfb54db9e31390c366b361ca18a34a37f8208e447a0ad3dc544226d96"} Nov 02 23:00:41.097776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.097703 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" event={"ID":"c5e6a984-3db4-4e54-93da-f857384f5316","Type":"ContainerStarted","Data":"13c23cf68448238080c5ee02779af852902448bb7368669c0b5eff597aba6b0f"} Nov 02 23:00:41.097776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.097753 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:41.237249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.236358 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" podStartSLOduration=1.236343349 podStartE2EDuration="1.236343349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:41.234139539 +0000 UTC m=+755.664009400" watchObservedRunningTime="2026-11-02 23:00:41.236343349 +0000 UTC m=+755.666213219" Nov 02 23:00:41.412775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.411236 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:00:41.412945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:41.412839 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:00:41.442696 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:00:41.442656 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732e98ae_cc72_48a1_b61f_c31f53b32bd3.slice/crio-e25d381a4b4eed5c712d69f8534bcb17b5373ccd56979d9de8d2a50ac640af42 WatchSource:0}: Error finding container e25d381a4b4eed5c712d69f8534bcb17b5373ccd56979d9de8d2a50ac640af42: Status 404 returned error can't find the container with id e25d381a4b4eed5c712d69f8534bcb17b5373ccd56979d9de8d2a50ac640af42 Nov 02 23:00:42.105085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:42.105045 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" event={"ID":"732e98ae-cc72-48a1-b61f-c31f53b32bd3","Type":"ContainerStarted","Data":"92e37d09c51b284fed5cbbbf469630a372d3f690230bc31ef06ae36c9cf45fad"} Nov 02 23:00:42.105622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:42.105105 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" event={"ID":"732e98ae-cc72-48a1-b61f-c31f53b32bd3","Type":"ContainerStarted","Data":"e25d381a4b4eed5c712d69f8534bcb17b5373ccd56979d9de8d2a50ac640af42"} Nov 02 23:00:42.223247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:42.223166 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" podStartSLOduration=2.223147111 podStartE2EDuration="2.223147111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:42.22216563 +0000 UTC m=+756.652035488" watchObservedRunningTime="2026-11-02 23:00:42.223147111 +0000 UTC m=+756.653016974" Nov 02 23:00:43.110013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:43.109982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:44.156017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:44.155979 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:44.328385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:44.328361 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:44.524109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:44.524081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:44.913699 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:44.913659 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:44.913881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:44.913723 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:45.121757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.121722 4494 generic.go:356] "Generic (PLEG): container finished" podID="3f388740-7f65-414f-ac27-4feba5a816c4" containerID="88afd1adfae64cfdc78c9a648295f86110879dac050989f98b4f21a512f6843f" exitCode=0 Nov 02 23:00:45.121925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.121801 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerDied","Data":"88afd1adfae64cfdc78c9a648295f86110879dac050989f98b4f21a512f6843f"} Nov 02 23:00:45.122210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.122191 4494 scope.go:122] "RemoveContainer" containerID="88afd1adfae64cfdc78c9a648295f86110879dac050989f98b4f21a512f6843f" Nov 02 23:00:45.123486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.123458 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:45.123601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.123554 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:45.123666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:45.123603 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:46.078983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:46.078921 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:00:46.129376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:46.129340 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerStarted","Data":"c0017828fc8502ad3517affb89fb94299cb8dbfa0b759564ec57f20706fa50f8"} Nov 02 23:00:46.189798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:46.189754 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:46.189981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:46.189835 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:46.189981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:46.189876 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:47.189863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:47.189828 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:47.190353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:47.189883 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:47.597194 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:00:47.597098 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:00:48.189602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:48.189563 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:48.189779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:48.189617 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:49.189466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:49.189426 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:49.189886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:49.189513 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:49.418816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:49.418780 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:49.582017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:49.581913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:49.775990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:49.775958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:50.190087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:50.190051 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:50.190507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:50.190114 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:51.189450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:51.189407 4494 patch_prober.go:29] interesting pod/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 23:00:51.189638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:51.189494 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 23:00:52.108781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.108700 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:00:52.181790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.181756 4494 generic.go:356] "Generic (PLEG): container finished" podID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerID="4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655" exitCode=0 Nov 02 23:00:52.181961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.181828 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerDied","Data":"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655"} Nov 02 23:00:52.182256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.182241 4494 scope.go:122] "RemoveContainer" containerID="4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655" Nov 02 23:00:52.183638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.183605 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:52.183739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.183683 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:52.183739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.183715 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:00:52.194373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.194351 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:52.194641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.194625 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:52.195412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:52.195398 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:00:53.190536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:53.190498 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerStarted","Data":"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9"} Nov 02 23:00:53.803719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:53.803686 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:53.803911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:53.803864 4494 patch_prober.go:29] interesting pod/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:00:53.803960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:53.803915 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 23:00:54.118562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:54.118491 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:00:54.760264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:54.756640 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:54.804184 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:54.804137 4494 patch_prober.go:29] interesting pod/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:00:54.804448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:54.804420 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 23:00:55.803773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:55.803728 4494 patch_prober.go:29] interesting pod/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:00:55.804303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:55.803791 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 23:00:56.803437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:56.803403 4494 patch_prober.go:29] interesting pod/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:00:56.803626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:56.803470 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 23:00:57.803411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:57.803371 4494 patch_prober.go:29] interesting pod/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 23:00:57.803892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:57.803482 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 23:00:58.865751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:58.865451 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:58.866429 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:58.866404 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:58.867992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:58.867969 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:00:59.367552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:59.367511 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:00:59.543497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:00:59.543459 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:00.068934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:00.068893 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:00.916738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:00.915888 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:01:00.916738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:00.915960 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:01:01.241234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.241128 4494 generic.go:356] "Generic (PLEG): container finished" podID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerID="8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3" exitCode=0 Nov 02 23:01:01.241234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.241208 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerDied","Data":"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3"} Nov 02 23:01:01.241671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.241652 4494 scope.go:122] "RemoveContainer" containerID="8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3" Nov 02 23:01:01.243105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.243077 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:01.243188 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.243146 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:01.243229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:01.243192 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:02.259122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.259080 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerStarted","Data":"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9"} Nov 02 23:01:02.796109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.796075 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb"] Nov 02 23:01:02.799969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.799952 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:02.851576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.851543 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:02.851908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.851885 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 23:01:02.852072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.852046 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 23:01:02.856195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.855512 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:01:02.856195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.855629 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:01:02.859271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.859246 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb"] Nov 02 23:01:02.906645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.906608 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thws8\" (UniqueName: \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:02.906836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.906648 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:02.906836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:02.906739 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.007941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.007911 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thws8\" (UniqueName: \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.008139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.007948 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.008139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.008008 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.008824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.008799 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.010723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.010703 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.069870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.069794 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thws8\" (UniqueName: \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\") pod \"maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.113043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.113013 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:03.516423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.516397 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb"] Nov 02 23:01:03.518280 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:03.518252 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8444ea07_e3a2_4494_abbd_d558692c9d9e.slice/crio-f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863 WatchSource:0}: Error finding container f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863: Status 404 returned error can't find the container with id f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863 Nov 02 23:01:03.853090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.852572 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 23:01:03.853090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:03.852636 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 23:01:04.270207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:04.270151 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" event={"ID":"8444ea07-e3a2-4494-abbd-d558692c9d9e","Type":"ContainerStarted","Data":"f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863"} Nov 02 23:01:04.841199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:04.840550 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 23:01:04.841199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:04.840609 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 23:01:05.483351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:05.483305 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:05.854695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:05.854584 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 23:01:05.855381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:05.855335 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 23:01:06.852124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:06.851628 4494 patch_prober.go:29] interesting pod/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 23:01:06.852124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:06.851696 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 23:01:07.843573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:07.843544 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:07.844079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:07.843956 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:07.844539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:07.844518 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:08.297062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:08.296977 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" event={"ID":"8444ea07-e3a2-4494-abbd-d558692c9d9e","Type":"ContainerStarted","Data":"a8678d7be255420b5cdb7b3624c048b89b3311b65a386987ad9f7aa86274a486"} Nov 02 23:01:08.417546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:08.417491 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" podStartSLOduration=2.316185659 podStartE2EDuration="6.417476272s" totalImagesPullingTime="4.101290613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:08.414462336 +0000 UTC m=+782.844332203" watchObservedRunningTime="2026-11-02 23:01:08.417476272 +0000 UTC m=+782.847346175" Nov 02 23:01:09.303135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:09.303095 4494 generic.go:356] "Generic (PLEG): container finished" podID="8444ea07-e3a2-4494-abbd-d558692c9d9e" containerID="a8678d7be255420b5cdb7b3624c048b89b3311b65a386987ad9f7aa86274a486" exitCode=0 Nov 02 23:01:09.303632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:09.303206 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" event={"ID":"8444ea07-e3a2-4494-abbd-d558692c9d9e","Type":"ContainerDied","Data":"a8678d7be255420b5cdb7b3624c048b89b3311b65a386987ad9f7aa86274a486"} Nov 02 23:01:10.447906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.447876 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:10.570871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.570774 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:10.616979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.616945 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\") pod \"8444ea07-e3a2-4494-abbd-d558692c9d9e\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " Nov 02 23:01:10.617187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.616995 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\") pod \"8444ea07-e3a2-4494-abbd-d558692c9d9e\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " Nov 02 23:01:10.617287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.617161 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\" (UniqueName: \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\") pod \"8444ea07-e3a2-4494-abbd-d558692c9d9e\" (UID: \"8444ea07-e3a2-4494-abbd-d558692c9d9e\") " Nov 02 23:01:10.617359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.617298 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca" pod "8444ea07-e3a2-4494-abbd-d558692c9d9e" (UID: "8444ea07-e3a2-4494-abbd-d558692c9d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:10.617625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.617597 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:10.619494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.619469 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8" pod "8444ea07-e3a2-4494-abbd-d558692c9d9e" (UID: "8444ea07-e3a2-4494-abbd-d558692c9d9e"). InnerVolumeSpecName "kube-api-access-thws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:10.619600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.619568 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca" pod "8444ea07-e3a2-4494-abbd-d558692c9d9e" (UID: "8444ea07-e3a2-4494-abbd-d558692c9d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:10.718956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.718925 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8444ea07-e3a2-4494-abbd-d558692c9d9e-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:10.718956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.718953 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thws8\" (UniqueName: \"kubernetes.io/projected/8444ea07-e3a2-4494-abbd-d558692c9d9e-kube-api-access-thws8\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:10.962193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.962136 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp"] Nov 02 23:01:10.962735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.962713 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8444ea07-e3a2-4494-abbd-d558692c9d9e" containerName="revoke-keys" Nov 02 23:01:10.962848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.962739 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8444ea07-e3a2-4494-abbd-d558692c9d9e" containerName="revoke-keys" Nov 02 23:01:10.962936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.962919 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8444ea07-e3a2-4494-abbd-d558692c9d9e" containerName="revoke-keys" Nov 02 23:01:10.965972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:10.965941 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.022930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.022891 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp"] Nov 02 23:01:11.122964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.122931 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.123192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.123015 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.123192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.123111 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv96f\" (UniqueName: \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.224480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.224390 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.224480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.224447 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.224740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.224566 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv96f\" (UniqueName: \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.225022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.225004 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.227124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.227097 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.286245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.286216 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv96f\" (UniqueName: \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\") pod \"maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.314352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.314323 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" Nov 02 23:01:11.314489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.314359 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb" event={"ID":"8444ea07-e3a2-4494-abbd-d558692c9d9e","Type":"ContainerDied","Data":"f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863"} Nov 02 23:01:11.314489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.314413 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5a1c67aa5b02f888139826e9d130526437b69f61be8dd12b0dc75c04f31b863" Nov 02 23:01:11.581662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.581577 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:11.960444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:11.960419 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp"] Nov 02 23:01:11.961705 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:11.961675 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fa7bfd_03d2_4d1c_b154_431e04e41224.slice/crio-9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500 WatchSource:0}: Error finding container 9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500: Status 404 returned error can't find the container with id 9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500 Nov 02 23:01:12.321348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:12.321313 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" event={"ID":"90fa7bfd-03d2-4d1c-b154-431e04e41224","Type":"ContainerStarted","Data":"b2c4922450639e6976ba83b376b2ba93bdd64f170c408374321171b633d7511a"} Nov 02 23:01:12.321561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:12.321405 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" event={"ID":"90fa7bfd-03d2-4d1c-b154-431e04e41224","Type":"ContainerStarted","Data":"9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500"} Nov 02 23:01:12.439710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:12.439655 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" podStartSLOduration=2.439640249 podStartE2EDuration="2.439640249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:12.438296106 +0000 UTC m=+786.868165967" watchObservedRunningTime="2026-11-02 23:01:12.439640249 +0000 UTC m=+786.869510116" Nov 02 23:01:13.328229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.328166 4494 generic.go:356] "Generic (PLEG): container finished" podID="90fa7bfd-03d2-4d1c-b154-431e04e41224" containerID="b2c4922450639e6976ba83b376b2ba93bdd64f170c408374321171b633d7511a" exitCode=0 Nov 02 23:01:13.328633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.328276 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" event={"ID":"90fa7bfd-03d2-4d1c-b154-431e04e41224","Type":"ContainerDied","Data":"b2c4922450639e6976ba83b376b2ba93bdd64f170c408374321171b633d7511a"} Nov 02 23:01:13.484708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.484679 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp"] Nov 02 23:01:13.488646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.488627 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.544076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.544039 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-785kn\" (UniqueName: \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.544312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.544284 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.544419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.544395 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.557502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.557473 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp"] Nov 02 23:01:13.645287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.645161 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-785kn\" (UniqueName: \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.645471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.645332 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.645471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.645393 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.646241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.646212 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.647966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.647942 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.706875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.706849 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-785kn\" (UniqueName: \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\") pod \"maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.717621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.717595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:13.803725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.803688 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:13.862889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:13.862858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:14.183378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.183345 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp"] Nov 02 23:01:14.184895 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:14.184867 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cd1ae4_eaf1_4912_a523_bad705cb5ff7.slice/crio-c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c WatchSource:0}: Error finding container c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c: Status 404 returned error can't find the container with id c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c Nov 02 23:01:14.335998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.335969 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" event={"ID":"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7","Type":"ContainerStarted","Data":"c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c"} Nov 02 23:01:14.474104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.474082 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:14.553994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.553954 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\") pod \"90fa7bfd-03d2-4d1c-b154-431e04e41224\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " Nov 02 23:01:14.554156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.554029 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\" (UniqueName: \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\") pod \"90fa7bfd-03d2-4d1c-b154-431e04e41224\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " Nov 02 23:01:14.554156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.554137 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\") pod \"90fa7bfd-03d2-4d1c-b154-431e04e41224\" (UID: \"90fa7bfd-03d2-4d1c-b154-431e04e41224\") " Nov 02 23:01:14.554386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.554361 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca" pod "90fa7bfd-03d2-4d1c-b154-431e04e41224" (UID: "90fa7bfd-03d2-4d1c-b154-431e04e41224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:14.554551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.554535 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:14.556140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.556121 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f" pod "90fa7bfd-03d2-4d1c-b154-431e04e41224" (UID: "90fa7bfd-03d2-4d1c-b154-431e04e41224"). InnerVolumeSpecName "kube-api-access-pv96f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:14.556386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.556364 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca" pod "90fa7bfd-03d2-4d1c-b154-431e04e41224" (UID: "90fa7bfd-03d2-4d1c-b154-431e04e41224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:14.656098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.656061 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv96f\" (UniqueName: \"kubernetes.io/projected/90fa7bfd-03d2-4d1c-b154-431e04e41224-kube-api-access-pv96f\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:14.656098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:14.656094 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90fa7bfd-03d2-4d1c-b154-431e04e41224-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:15.343474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.343389 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" Nov 02 23:01:15.343938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.343392 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp" event={"ID":"90fa7bfd-03d2-4d1c-b154-431e04e41224","Type":"ContainerDied","Data":"9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500"} Nov 02 23:01:15.343938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.343538 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c9c8b5a498a03d8f6f0129af9a1f9d6fac8db1012fe26e07b903289a790e500" Nov 02 23:01:15.345208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.345161 4494 generic.go:356] "Generic (PLEG): container finished" podID="d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" containerID="f9f9c6949616d0c3f5af722ce90e5bd999a2fee7aacd070ca6988e1e0be8bc5c" exitCode=0 Nov 02 23:01:15.345372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.345209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" event={"ID":"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7","Type":"ContainerDied","Data":"f9f9c6949616d0c3f5af722ce90e5bd999a2fee7aacd070ca6988e1e0be8bc5c"} Nov 02 23:01:15.975027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:15.974980 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:16.489217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.489192 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:16.575894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.575862 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\") pod \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " Nov 02 23:01:16.576110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.575924 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\") pod \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " Nov 02 23:01:16.576110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.575997 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\" (UniqueName: \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\") pod \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\" (UID: \"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7\") " Nov 02 23:01:16.576287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.576205 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca" pod "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" (UID: "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:16.576447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.576428 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:16.578206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.578161 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca" pod "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" (UID: "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:16.578301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.578236 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn" pod "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" (UID: "d0cd1ae4-eaf1-4912-a523-bad705cb5ff7"). InnerVolumeSpecName "kube-api-access-785kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:16.677713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.677683 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:16.677859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:16.677715 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-785kn\" (UniqueName: \"kubernetes.io/projected/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7-kube-api-access-785kn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:17.357248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:17.357217 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" Nov 02 23:01:17.357248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:17.357238 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp" event={"ID":"d0cd1ae4-eaf1-4912-a523-bad705cb5ff7","Type":"ContainerDied","Data":"c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c"} Nov 02 23:01:17.357544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:17.357286 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e90662f2f39c5258514db0a0cf9c6458b70a7accd1037bf3252885e87c264c" Nov 02 23:01:17.558226 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:17.558161 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:01:21.346431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:21.346400 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:22.861316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:22.861282 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:01:23.226092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.226020 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:23.397602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.397566 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:23.555677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.555594 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq"] Nov 02 23:01:23.556041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556028 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" containerName="revoke-keys" Nov 02 23:01:23.556092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556042 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" containerName="revoke-keys" Nov 02 23:01:23.556092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556065 4494 cpu_manager.go:414] "containerMap: removing container" podUID="90fa7bfd-03d2-4d1c-b154-431e04e41224" containerName="revoke-keys" Nov 02 23:01:23.556092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556070 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fa7bfd-03d2-4d1c-b154-431e04e41224" containerName="revoke-keys" Nov 02 23:01:23.556216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556147 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" containerName="revoke-keys" Nov 02 23:01:23.556216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.556158 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fa7bfd-03d2-4d1c-b154-431e04e41224" containerName="revoke-keys" Nov 02 23:01:23.559301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.559286 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.615215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.615140 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:01:23.615432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.615315 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:01:23.618319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.618296 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq"] Nov 02 23:01:23.646849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.646817 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.646971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.646920 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfz6d\" (UniqueName: \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.647096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.647070 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.748462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.748431 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.748654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.748504 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.748654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.748534 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz6d\" (UniqueName: \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.749118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.749092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.750903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.750886 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.811042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.810977 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz6d\" (UniqueName: \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\") pod \"maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:23.875918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:23.875889 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:24.249546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:24.249522 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq"] Nov 02 23:01:24.250612 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:24.250583 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eb5143_01f5_43a3_81f5_fc8b4b5c329f.slice/crio-72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655 WatchSource:0}: Error finding container 72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655: Status 404 returned error can't find the container with id 72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655 Nov 02 23:01:24.393661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:24.393622 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" event={"ID":"75eb5143-01f5-43a3-81f5-fc8b4b5c329f","Type":"ContainerStarted","Data":"72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655"} Nov 02 23:01:25.400698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:25.400661 4494 generic.go:356] "Generic (PLEG): container finished" podID="75eb5143-01f5-43a3-81f5-fc8b4b5c329f" containerID="b5aadd8ae01da3ae0a6ab4eed93177d16e97789c6053a405e01b46856f4b04f1" exitCode=0 Nov 02 23:01:25.401096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:25.400750 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" event={"ID":"75eb5143-01f5-43a3-81f5-fc8b4b5c329f","Type":"ContainerDied","Data":"b5aadd8ae01da3ae0a6ab4eed93177d16e97789c6053a405e01b46856f4b04f1"} Nov 02 23:01:25.666399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:25.666361 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:25.814710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:25.814674 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:25.963312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:25.963225 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:26.570456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.570427 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:26.681808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.681781 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\" (UniqueName: \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\") pod \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " Nov 02 23:01:26.681976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.681860 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\") pod \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " Nov 02 23:01:26.681976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.681919 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\") pod \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\" (UID: \"75eb5143-01f5-43a3-81f5-fc8b4b5c329f\") " Nov 02 23:01:26.682292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.682267 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca" pod "75eb5143-01f5-43a3-81f5-fc8b4b5c329f" (UID: "75eb5143-01f5-43a3-81f5-fc8b4b5c329f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:26.682395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.682378 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.683988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.683949 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d" pod "75eb5143-01f5-43a3-81f5-fc8b4b5c329f" (UID: "75eb5143-01f5-43a3-81f5-fc8b4b5c329f"). InnerVolumeSpecName "kube-api-access-bfz6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:26.684070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.684011 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca" pod "75eb5143-01f5-43a3-81f5-fc8b4b5c329f" (UID: "75eb5143-01f5-43a3-81f5-fc8b4b5c329f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:26.783556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.783522 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:26.783556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:26.783552 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfz6d\" (UniqueName: \"kubernetes.io/projected/75eb5143-01f5-43a3-81f5-fc8b4b5c329f-kube-api-access-bfz6d\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:27.243777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.243743 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:27.411724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.411698 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" Nov 02 23:01:27.411724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.411709 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq" event={"ID":"75eb5143-01f5-43a3-81f5-fc8b4b5c329f","Type":"ContainerDied","Data":"72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655"} Nov 02 23:01:27.411969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.411751 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72d40767a57c63f21e3c5f989a2f6f5a10d393c6e7526e21283bf206f0620655" Nov 02 23:01:27.612368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.612287 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:27.862612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.862515 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" podUID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" containerName="payload-processing" containerID="cri-o://4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071" gracePeriod=25 Nov 02 23:01:27.902515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:27.902486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:28.126046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.125984 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:01:28.196731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.196701 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\" (UniqueName: \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\") pod \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " Nov 02 23:01:28.196940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.196780 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\") pod \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\" (UID: \"1b05de54-35e1-44b0-b5b4-ca49e4df3c15\") " Nov 02 23:01:28.197147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.197123 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume" pod "1b05de54-35e1-44b0-b5b4-ca49e4df3c15" (UID: "1b05de54-35e1-44b0-b5b4-ca49e4df3c15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:28.200895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.200868 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr" pod "1b05de54-35e1-44b0-b5b4-ca49e4df3c15" (UID: "1b05de54-35e1-44b0-b5b4-ca49e4df3c15"). InnerVolumeSpecName "kube-api-access-26pvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:28.203721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.203695 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb"] Nov 02 23:01:28.211711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.211688 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d7bf49f-z7gxb"] Nov 02 23:01:28.230950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.230920 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:28.254355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.254329 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:01:28.254524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.254506 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" containerID="cri-o://c0017828fc8502ad3517affb89fb94299cb8dbfa0b759564ec57f20706fa50f8" gracePeriod=30 Nov 02 23:01:28.328676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.328632 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pvr\" (UniqueName: \"kubernetes.io/projected/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-kube-api-access-26pvr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.328676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.328677 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b05de54-35e1-44b0-b5b4-ca49e4df3c15-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:28.418030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.417996 4494 generic.go:356] "Generic (PLEG): container finished" podID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" containerID="4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071" exitCode=0 Nov 02 23:01:28.418278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.418053 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" event={"ID":"1b05de54-35e1-44b0-b5b4-ca49e4df3c15","Type":"ContainerDied","Data":"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071"} Nov 02 23:01:28.418278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.418083 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" Nov 02 23:01:28.418278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.418106 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5" event={"ID":"1b05de54-35e1-44b0-b5b4-ca49e4df3c15","Type":"ContainerDied","Data":"21bfac74deda43ed0964a8dc3b1731cba718c2ff971e9af053ec16e9a279a1e0"} Nov 02 23:01:28.418278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.418130 4494 scope.go:122] "RemoveContainer" containerID="4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071" Nov 02 23:01:28.427879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.427864 4494 scope.go:122] "RemoveContainer" containerID="4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071" Nov 02 23:01:28.428133 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:28.428115 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071\": container with ID starting with 4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071 not found: ID does not exist" containerID="4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071" Nov 02 23:01:28.428206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.428141 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071"} err="failed to get container status \"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071\": rpc error: code = NotFound desc = could not find container \"4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071\": container with ID starting with 4b5a41638e4241b98a307d0bbacf9cbcb97b93f851b8841b1790ef048077d071 not found: ID does not exist" Nov 02 23:01:28.537920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.537888 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:28.587737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.587710 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:01:28.591024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.591002 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5dd28258-84887df7f5-vd9v5"] Nov 02 23:01:28.894354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:28.894267 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:29.043162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.043126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:29.191469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.191441 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:29.424013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.423983 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp"] Nov 02 23:01:29.477083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.476992 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5dd28258-qtxzp"] Nov 02 23:01:29.615842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.615809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:29.695294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.695260 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" path="/var/lib/kubelet/pods/1b05de54-35e1-44b0-b5b4-ca49e4df3c15/volumes" Nov 02 23:01:29.695804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.695790 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8444ea07-e3a2-4494-abbd-d558692c9d9e" path="/var/lib/kubelet/pods/8444ea07-e3a2-4494-abbd-d558692c9d9e/volumes" Nov 02 23:01:29.696289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.696275 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fa7bfd-03d2-4d1c-b154-431e04e41224" path="/var/lib/kubelet/pods/90fa7bfd-03d2-4d1c-b154-431e04e41224/volumes" Nov 02 23:01:29.977103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:29.977069 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:30.125260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.125229 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:30.431119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.431080 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:30.578388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.578359 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:30.705309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.705228 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp"] Nov 02 23:01:30.756077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.756040 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed9e8ee-jbgzp"] Nov 02 23:01:30.781210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.781165 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:01:30.781456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.781429 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" containerID="cri-o://7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9" gracePeriod=30 Nov 02 23:01:30.807778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:30.807703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.054477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.054400 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.103201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103150 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7"] Nov 02 23:01:31.103624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103606 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" containerName="payload-processing" Nov 02 23:01:31.103667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103625 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" containerName="payload-processing" Nov 02 23:01:31.103667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103644 4494 cpu_manager.go:414] "containerMap: removing container" podUID="75eb5143-01f5-43a3-81f5-fc8b4b5c329f" containerName="revoke-keys" Nov 02 23:01:31.103667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103653 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75eb5143-01f5-43a3-81f5-fc8b4b5c329f" containerName="revoke-keys" Nov 02 23:01:31.103759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103730 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b05de54-35e1-44b0-b5b4-ca49e4df3c15" containerName="payload-processing" Nov 02 23:01:31.103759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.103739 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="75eb5143-01f5-43a3-81f5-fc8b4b5c329f" containerName="revoke-keys" Nov 02 23:01:31.108278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.108263 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.163254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.163223 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:01:31.163438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.163413 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:01:31.166638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.166615 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7"] Nov 02 23:01:31.262694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.262657 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.262884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.262790 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpkv\" (UniqueName: \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.262884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.262834 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.322049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.321962 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 02 23:01:31.325709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.325687 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.364587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.364538 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.364772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.364665 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.364842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.364781 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpkv\" (UniqueName: \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.365697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.365667 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.379267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.379239 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.389997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.389966 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 02 23:01:31.412695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.412663 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ec2f9a-openshift-default-dockercfg-4nt5j\"" Nov 02 23:01:31.418622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.418594 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.426711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.426676 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpkv\" (UniqueName: \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\") pod \"maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.465998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmsg\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466122 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466165 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466244 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466290 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466340 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466369 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466492 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.470384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.466527 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.568762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.568718 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.568762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.568761 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.568864 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmsg\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.568920 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.568950 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569001 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569045 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569091 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569119 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569371 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569485 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569518 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569550 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.569739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.569713 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.571951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.571926 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.572041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.572003 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.582338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.582271 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.630646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.630617 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.630797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.630720 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmsg\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.638065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.638048 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:31.695942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.695673 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cd1ae4-eaf1-4912-a523-bad705cb5ff7" path="/var/lib/kubelet/pods/d0cd1ae4-eaf1-4912-a523-bad705cb5ff7/volumes" Nov 02 23:01:31.721490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.721161 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:31.732386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:31.732359 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:32.072572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.072550 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 02 23:01:32.075165 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:32.075126 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a87125_f959_4252_8e96_ede7e8de6a14.slice/crio-398ffb0db3813b6645ec4985b1fa83bb165267233dc7a7689dade61dc3150fcb WatchSource:0}: Error finding container 398ffb0db3813b6645ec4985b1fa83bb165267233dc7a7689dade61dc3150fcb: Status 404 returned error can't find the container with id 398ffb0db3813b6645ec4985b1fa83bb165267233dc7a7689dade61dc3150fcb Nov 02 23:01:32.077804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.077774 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.077928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.077833 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.077928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.077863 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:32.121396 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:32.121365 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e75b5a_d2c5_4741_8270_6a982a0f6e87.slice/crio-0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345 WatchSource:0}: Error finding container 0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345: Status 404 returned error can't find the container with id 0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345 Nov 02 23:01:32.121716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.121699 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7"] Nov 02 23:01:32.340816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.336931 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:32.454659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.453780 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" event={"ID":"39e75b5a-d2c5-4741-8270-6a982a0f6e87","Type":"ContainerStarted","Data":"0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345"} Nov 02 23:01:32.456595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.456121 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerStarted","Data":"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948"} Nov 02 23:01:32.456719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.456623 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerStarted","Data":"398ffb0db3813b6645ec4985b1fa83bb165267233dc7a7689dade61dc3150fcb"} Nov 02 23:01:32.578996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.578944 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podStartSLOduration=1.578932175 podStartE2EDuration="1.578932175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:32.578552572 +0000 UTC m=+807.008422433" watchObservedRunningTime="2026-11-02 23:01:32.578932175 +0000 UTC m=+807.008802037" Nov 02 23:01:32.609238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.609150 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:32.638523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:32.638499 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:33.072889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.072855 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:01:33.465405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.465375 4494 generic.go:356] "Generic (PLEG): container finished" podID="3f388740-7f65-414f-ac27-4feba5a816c4" containerID="c0017828fc8502ad3517affb89fb94299cb8dbfa0b759564ec57f20706fa50f8" exitCode=0 Nov 02 23:01:33.465588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.465453 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerDied","Data":"c0017828fc8502ad3517affb89fb94299cb8dbfa0b759564ec57f20706fa50f8"} Nov 02 23:01:33.465588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.465532 4494 scope.go:122] "RemoveContainer" containerID="88afd1adfae64cfdc78c9a648295f86110879dac050989f98b4f21a512f6843f" Nov 02 23:01:33.467500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.467473 4494 generic.go:356] "Generic (PLEG): container finished" podID="39e75b5a-d2c5-4741-8270-6a982a0f6e87" containerID="6cad3909c3928b684b740671305a03e0625845a11bb3b0e7a8048fd83f7f80e2" exitCode=0 Nov 02 23:01:33.467698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.467548 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" event={"ID":"39e75b5a-d2c5-4741-8270-6a982a0f6e87","Type":"ContainerDied","Data":"6cad3909c3928b684b740671305a03e0625845a11bb3b0e7a8048fd83f7f80e2"} Nov 02 23:01:33.513754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.513733 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:01:33.592763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592679 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.592763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592723 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.592963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592799 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.592963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592868 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.592963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592906 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.592963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592941 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.593165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.592981 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:33.593165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593017 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.593165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593058 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.593165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593104 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\") pod \"3f388740-7f65-414f-ac27-4feba5a816c4\" (UID: \"3f388740-7f65-414f-ac27-4feba5a816c4\") " Nov 02 23:01:33.593393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593223 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:33.593393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593372 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:33.593823 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593804 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f388740-7f65-414f-ac27-4feba5a816c4-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.593929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593827 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.593929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593841 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.594063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.593939 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:33.594063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.594009 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:33.595441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.595412 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:33.595441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.595429 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:01:33.595689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.595669 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:33.595769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.595744 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln" pod "3f388740-7f65-414f-ac27-4feba5a816c4" (UID: "3f388740-7f65-414f-ac27-4feba5a816c4"). InnerVolumeSpecName "kube-api-access-8xnln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:33.644726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.644702 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:33.695565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695534 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.695713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695570 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f388740-7f65-414f-ac27-4feba5a816c4-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.695713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695586 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.695713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695600 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnln\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-kube-api-access-8xnln\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.695713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695616 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f388740-7f65-414f-ac27-4feba5a816c4-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:33.695713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:33.695630 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f388740-7f65-414f-ac27-4feba5a816c4-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:34.228899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.228837 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:34.416977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.416929 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:34.474672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.474635 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" Nov 02 23:01:34.474852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.474674 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86" event={"ID":"3f388740-7f65-414f-ac27-4feba5a816c4","Type":"ContainerDied","Data":"72d2c78c73b6bafcf0f3594bcdcb4afb76b1a5549f9735bd790be2427bbb4e96"} Nov 02 23:01:34.474852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.474733 4494 scope.go:122] "RemoveContainer" containerID="c0017828fc8502ad3517affb89fb94299cb8dbfa0b759564ec57f20706fa50f8" Nov 02 23:01:34.475653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.475328 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:34.476621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.476590 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:01:34.566548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.566502 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:34.619001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.618974 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:34.706132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.706099 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\" (UniqueName: \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\") pod \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " Nov 02 23:01:34.706340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.706157 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\") pod \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " Nov 02 23:01:34.706340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.706306 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\") pod \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\" (UID: \"39e75b5a-d2c5-4741-8270-6a982a0f6e87\") " Nov 02 23:01:34.706653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.706624 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca" pod "39e75b5a-d2c5-4741-8270-6a982a0f6e87" (UID: "39e75b5a-d2c5-4741-8270-6a982a0f6e87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:34.708687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.708660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv" pod "39e75b5a-d2c5-4741-8270-6a982a0f6e87" (UID: "39e75b5a-d2c5-4741-8270-6a982a0f6e87"). InnerVolumeSpecName "kube-api-access-gxpkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:34.708687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.708672 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca" pod "39e75b5a-d2c5-4741-8270-6a982a0f6e87" (UID: "39e75b5a-d2c5-4741-8270-6a982a0f6e87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:01:34.808670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.808578 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:34.808670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.808616 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxpkv\" (UniqueName: \"kubernetes.io/projected/39e75b5a-d2c5-4741-8270-6a982a0f6e87-kube-api-access-gxpkv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:34.808670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.808631 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e75b5a-d2c5-4741-8270-6a982a0f6e87-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:34.815761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.815728 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:01:34.818135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:34.818109 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7d7bf49f-openshift-default-5685cd6d99-ccx86"] Nov 02 23:01:35.234845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.234794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:35.484049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.484018 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" Nov 02 23:01:35.484049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.484023 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7" event={"ID":"39e75b5a-d2c5-4741-8270-6a982a0f6e87","Type":"ContainerDied","Data":"0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345"} Nov 02 23:01:35.484274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.484083 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efd0da5ea67df738e62f3453f80444d15cc4578a6e87f696c21dab0980e7345" Nov 02 23:01:35.543848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.543809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:35.689622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.689589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:35.692559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:35.692536 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" path="/var/lib/kubelet/pods/3f388740-7f65-414f-ac27-4feba5a816c4/volumes" Nov 02 23:01:36.062063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.061999 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:01:36.222678 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222645 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.222857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222694 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.222857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222720 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.222857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222750 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.222857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222791 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.222857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222848 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.223120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222931 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.223120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.222977 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.223120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223007 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\") pod \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\" (UID: \"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8\") " Nov 02 23:01:36.223120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223104 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:36.223359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223116 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:36.223359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223324 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:36.223359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223337 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:36.223717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223693 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.223822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223723 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.223822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223739 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.223822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223754 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.224148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.223830 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:36.225499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.225477 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:36.226556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.226535 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "kube-api-access-622xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:36.227207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.227059 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:36.227207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.227148 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo" pod "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" (UID: "5c115ebb-fcf9-4bfb-8667-f72f4a933fa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:01:36.324954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.324871 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.324954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.324902 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.324954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.324912 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.324954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.324922 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-622xp\" (UniqueName: \"kubernetes.io/projected/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-kube-api-access-622xp\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.324954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.324932 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:36.493408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.493375 4494 generic.go:356] "Generic (PLEG): container finished" podID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerID="7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9" exitCode=0 Nov 02 23:01:36.493576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.493449 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" Nov 02 23:01:36.493576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.493460 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerDied","Data":"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9"} Nov 02 23:01:36.493576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.493537 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5" event={"ID":"5c115ebb-fcf9-4bfb-8667-f72f4a933fa8","Type":"ContainerDied","Data":"5249bdf3590977ac02c331c7a4c7c2babbe07df486d2a97852b191156dae21e2"} Nov 02 23:01:36.493576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.493559 4494 scope.go:122] "RemoveContainer" containerID="7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9" Nov 02 23:01:36.503361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.503343 4494 scope.go:122] "RemoveContainer" containerID="4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655" Nov 02 23:01:36.511404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.511387 4494 scope.go:122] "RemoveContainer" containerID="7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9" Nov 02 23:01:36.511675 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:36.511656 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9\": container with ID starting with 7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9 not found: ID does not exist" containerID="7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9" Nov 02 23:01:36.511721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.511683 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9"} err="failed to get container status \"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9\": rpc error: code = NotFound desc = could not find container \"7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9\": container with ID starting with 7e0ec05a1bf6f00e42681e7b3b030d3cfb667ef3d9d8c19ec55146837c2f36d9 not found: ID does not exist" Nov 02 23:01:36.511721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.511700 4494 scope.go:122] "RemoveContainer" containerID="4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655" Nov 02 23:01:36.511933 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:36.511913 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655\": container with ID starting with 4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655 not found: ID does not exist" containerID="4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655" Nov 02 23:01:36.511973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.511940 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655"} err="failed to get container status \"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655\": rpc error: code = NotFound desc = could not find container \"4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655\": container with ID starting with 4e3ff2f55fa29ee8a83216cad9a1d9659c62a2af61e4af9011eb84600e1ab655 not found: ID does not exist" Nov 02 23:01:36.660881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.660857 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:01:36.711956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:36.711930 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fed9e8ee-gw-openshift-default-c7749df7c-xndf5"] Nov 02 23:01:37.005387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.005303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:37.172912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.172879 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:37.328361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.328272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:37.658493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.658420 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:37.693598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.693573 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" path="/var/lib/kubelet/pods/5c115ebb-fcf9-4bfb-8667-f72f4a933fa8/volumes" Nov 02 23:01:37.824680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.824649 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:37.974928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:37.974898 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:38.774182 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:38.774134 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:38.964360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:38.964331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:42.690540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:42.690504 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:42.835039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:42.835001 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:43.103354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:43.103276 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:43.248691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:43.248655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:43.368252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:43.368162 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq"] Nov 02 23:01:43.418792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:43.418758 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f779f4ae-dlhhq"] Nov 02 23:01:43.692287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:43.692255 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75eb5143-01f5-43a3-81f5-fc8b4b5c329f" path="/var/lib/kubelet/pods/75eb5143-01f5-43a3-81f5-fc8b4b5c329f/volumes" Nov 02 23:01:44.866544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:44.866510 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:01:44.867017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:44.866772 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" containerID="cri-o://5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9" gracePeriod=30 Nov 02 23:01:44.877538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:44.877515 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:45.684215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:45.684150 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:46.200300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200266 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:01:46.200735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200722 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:46.200779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200736 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:46.200779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200745 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.200779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200750 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.200779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200767 4494 cpu_manager.go:414] "containerMap: removing container" podUID="39e75b5a-d2c5-4741-8270-6a982a0f6e87" containerName="revoke-keys" Nov 02 23:01:46.200779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200773 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e75b5a-d2c5-4741-8270-6a982a0f6e87" containerName="revoke-keys" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200782 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200786 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200862 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200869 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200877 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e75b5a-d2c5-4741-8270-6a982a0f6e87" containerName="revoke-keys" Nov 02 23:01:46.200933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.200883 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c115ebb-fcf9-4bfb-8667-f72f4a933fa8" containerName="istio-proxy" Nov 02 23:01:46.207584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.207563 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228165 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hlg\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228255 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228300 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228333 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228373 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228407 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228433 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228460 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.228655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.228538 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.264661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.264614 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-637fdb-openshift-default-dockercfg-brzsm\"" Nov 02 23:01:46.294144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.294108 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:46.317410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.317373 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:01:46.329849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.329809 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.329914 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.329962 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330007 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330064 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330118 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330145 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330304 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330371 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hlg\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.330747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330677 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.331028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.330997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.331132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.331062 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.331323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.331303 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.331406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.331341 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.333609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.333579 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.333979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.333943 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.392419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.392391 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.392552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.392433 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hlg\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.454580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.454430 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:46.520867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.520836 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:46.948064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.948039 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:01:46.949272 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:46.949247 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4625b18a_a96b_4208_b22d_cc49548112a0.slice/crio-d9c360bd85cf487de52ab3cde767f7f7db48ef1dee3b85c231e40f3346a48039 WatchSource:0}: Error finding container d9c360bd85cf487de52ab3cde767f7f7db48ef1dee3b85c231e40f3346a48039: Status 404 returned error can't find the container with id d9c360bd85cf487de52ab3cde767f7f7db48ef1dee3b85c231e40f3346a48039 Nov 02 23:01:46.951536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.951503 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:46.951608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.951568 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:46.951608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:46.951596 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:01:47.559376 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:47.559333 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:01:47.570886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:47.570851 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerStarted","Data":"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d"} Nov 02 23:01:47.571048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:47.570910 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerStarted","Data":"d9c360bd85cf487de52ab3cde767f7f7db48ef1dee3b85c231e40f3346a48039"} Nov 02 23:01:47.693669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:47.693615 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podStartSLOduration=1.693602602 podStartE2EDuration="1.693602602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:47.692214327 +0000 UTC m=+822.122084193" watchObservedRunningTime="2026-11-02 23:01:47.693602602 +0000 UTC m=+822.123472459" Nov 02 23:01:47.843267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:47.843159 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:48.016811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.016776 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:48.166406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.166361 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:48.253976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.253933 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 02 23:01:48.254408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.254394 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:48.254456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.254410 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:48.254513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.254504 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f388740-7f65-414f-ac27-4feba5a816c4" containerName="istio-proxy" Nov 02 23:01:48.257623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.257607 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.312794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.312763 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-96031a\"" Nov 02 23:01:48.313146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.313125 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-96031a-dockercfg-mkpdb\"" Nov 02 23:01:48.315026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.314998 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 02 23:01:48.317069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.317042 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:48.355412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.355158 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm2s\" (UniqueName: \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.356099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.356004 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.458791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.458702 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm2s\" (UniqueName: \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.458966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.458798 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.459505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.459485 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.520543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.520511 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm2s\" (UniqueName: \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\") pod \"payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.521264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.521246 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:48.526479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.526451 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:48.570842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.570810 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:48.577228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.577203 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:48.578145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.578125 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:01:48.587905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.587878 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:01:48.953295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:48.953268 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 02 23:01:48.954748 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:01:48.954718 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66492f3c_f750_466b_b051_74adddec8220.slice/crio-f583a3b18234657d58f50628946bd4b141796b0087c745bf15c9467879ec0c3c WatchSource:0}: Error finding container f583a3b18234657d58f50628946bd4b141796b0087c745bf15c9467879ec0c3c: Status 404 returned error can't find the container with id f583a3b18234657d58f50628946bd4b141796b0087c745bf15c9467879ec0c3c Nov 02 23:01:49.587236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:49.587199 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" event={"ID":"66492f3c-f750-466b-b051-74adddec8220","Type":"ContainerStarted","Data":"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e"} Nov 02 23:01:49.587721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:49.587259 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" event={"ID":"66492f3c-f750-466b-b051-74adddec8220","Type":"ContainerStarted","Data":"f583a3b18234657d58f50628946bd4b141796b0087c745bf15c9467879ec0c3c"} Nov 02 23:01:49.587721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:49.587304 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:01:49.704435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:49.704373 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" podStartSLOduration=1.7043559830000001 podStartE2EDuration="1.704355983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:49.703681283 +0000 UTC m=+824.133551144" watchObservedRunningTime="2026-11-02 23:01:49.704355983 +0000 UTC m=+824.134225846" Nov 02 23:01:50.129076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.129016 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:50.176411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176370 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176412 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176435 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176526 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176552 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176579 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176597 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176641 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.176666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176661 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\") pod \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\" (UID: \"6be0d79e-06f2-40ce-98d2-e8514a0c8895\") " Nov 02 23:01:50.177031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176733 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:50.177031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176761 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:50.177031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.176983 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:50.177213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.177037 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:50.177213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.177111 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.177213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.177130 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.177213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.177145 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.177425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.177242 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:50.178850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.178817 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:01:50.179130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.179098 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:01:50.179445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.179422 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:50.179521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.179439 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj" pod "6be0d79e-06f2-40ce-98d2-e8514a0c8895" (UID: "6be0d79e-06f2-40ce-98d2-e8514a0c8895"). InnerVolumeSpecName "kube-api-access-pt5cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:50.278194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278140 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.278194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278197 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.278402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278212 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5cj\" (UniqueName: \"kubernetes.io/projected/6be0d79e-06f2-40ce-98d2-e8514a0c8895-kube-api-access-pt5cj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.278402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278226 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.278402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278238 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.278402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.278249 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be0d79e-06f2-40ce-98d2-e8514a0c8895-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:50.595351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.595312 4494 generic.go:356] "Generic (PLEG): container finished" podID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerID="5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9" exitCode=0 Nov 02 23:01:50.595827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.595365 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" Nov 02 23:01:50.595827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.595387 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerDied","Data":"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9"} Nov 02 23:01:50.595827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.595435 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr" event={"ID":"6be0d79e-06f2-40ce-98d2-e8514a0c8895","Type":"ContainerDied","Data":"3fb964de45a0df2bf3e0ba85358217b852585e39bb7fb89ab240a5fc03f7d9e4"} Nov 02 23:01:50.595827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.595461 4494 scope.go:122] "RemoveContainer" containerID="5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9" Nov 02 23:01:50.604671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.604634 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:50.605529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.605515 4494 scope.go:122] "RemoveContainer" containerID="8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3" Nov 02 23:01:50.614529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.614511 4494 scope.go:122] "RemoveContainer" containerID="5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9" Nov 02 23:01:50.614781 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:50.614762 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9\": container with ID starting with 5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9 not found: ID does not exist" containerID="5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9" Nov 02 23:01:50.614827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.614791 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9"} err="failed to get container status \"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9\": rpc error: code = NotFound desc = could not find container \"5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9\": container with ID starting with 5511f3b44bed326ff70077a5c4d3ae22b2b10896fe8a325d32eb2b31b7509aa9 not found: ID does not exist" Nov 02 23:01:50.614827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.614810 4494 scope.go:122] "RemoveContainer" containerID="8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3" Nov 02 23:01:50.615053 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:01:50.615038 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3\": container with ID starting with 8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3 not found: ID does not exist" containerID="8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3" Nov 02 23:01:50.615091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.615059 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3"} err="failed to get container status \"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3\": rpc error: code = NotFound desc = could not find container \"8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3\": container with ID starting with 8baa29e339531fbb1fef896330053ec1738ce33ddff6d9f7e40a353cd75f28a3 not found: ID does not exist" Nov 02 23:01:50.767223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.767166 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:01:50.815365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.815323 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:50.817658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:50.817630 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f779f4ae-openshift-default-5cd5949c4c-dfbvr"] Nov 02 23:01:51.002144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.002110 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:51.299294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.299212 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:01:51.393982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.393944 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:01:51.650077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.649991 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:51.692325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.692298 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" path="/var/lib/kubelet/pods/6be0d79e-06f2-40ce-98d2-e8514a0c8895/volumes" Nov 02 23:01:51.799094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:51.799060 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:53.628523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:53.628485 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:53.782690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:53.782645 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:53.969515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:53.969481 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:56.299843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.299799 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" podUID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" containerName="payload-processing" containerID="cri-o://92e37d09c51b284fed5cbbbf469630a372d3f690230bc31ef06ae36c9cf45fad" gracePeriod=25 Nov 02 23:01:56.394295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.394248 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" podUID="c5e6a984-3db4-4e54-93da-f857384f5316" containerName="payload-pre-processing" containerID="cri-o://e971df1cfb54db9e31390c366b361ca18a34a37f8208e447a0ad3dc544226d96" gracePeriod=25 Nov 02 23:01:56.633989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.633933 4494 generic.go:356] "Generic (PLEG): container finished" podID="c5e6a984-3db4-4e54-93da-f857384f5316" containerID="e971df1cfb54db9e31390c366b361ca18a34a37f8208e447a0ad3dc544226d96" exitCode=0 Nov 02 23:01:56.634134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.634004 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" event={"ID":"c5e6a984-3db4-4e54-93da-f857384f5316","Type":"ContainerDied","Data":"e971df1cfb54db9e31390c366b361ca18a34a37f8208e447a0ad3dc544226d96"} Nov 02 23:01:56.636034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.635764 4494 generic.go:356] "Generic (PLEG): container finished" podID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" containerID="92e37d09c51b284fed5cbbbf469630a372d3f690230bc31ef06ae36c9cf45fad" exitCode=0 Nov 02 23:01:56.636034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.635802 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" event={"ID":"732e98ae-cc72-48a1-b61f-c31f53b32bd3","Type":"ContainerDied","Data":"92e37d09c51b284fed5cbbbf469630a372d3f690230bc31ef06ae36c9cf45fad"} Nov 02 23:01:56.653695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.653669 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:01:56.682053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.682028 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:01:56.746568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.746532 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\") pod \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " Nov 02 23:01:56.746775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.746600 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\" (UniqueName: \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\") pod \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\" (UID: \"732e98ae-cc72-48a1-b61f-c31f53b32bd3\") " Nov 02 23:01:56.746869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.746845 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\") pod \"c5e6a984-3db4-4e54-93da-f857384f5316\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " Nov 02 23:01:56.747948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.747921 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\" (UniqueName: \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\") pod \"c5e6a984-3db4-4e54-93da-f857384f5316\" (UID: \"c5e6a984-3db4-4e54-93da-f857384f5316\") " Nov 02 23:01:56.748818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.748759 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume" pod "c5e6a984-3db4-4e54-93da-f857384f5316" (UID: "c5e6a984-3db4-4e54-93da-f857384f5316"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:56.748818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.748757 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume" pod "732e98ae-cc72-48a1-b61f-c31f53b32bd3" (UID: "732e98ae-cc72-48a1-b61f-c31f53b32bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:01:56.750332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.749618 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732e98ae-cc72-48a1-b61f-c31f53b32bd3-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:56.750332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.749655 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e6a984-3db4-4e54-93da-f857384f5316-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:56.752709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.752672 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh" pod "732e98ae-cc72-48a1-b61f-c31f53b32bd3" (UID: "732e98ae-cc72-48a1-b61f-c31f53b32bd3"). InnerVolumeSpecName "kube-api-access-tzfxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:56.786722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.786681 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7" pod "c5e6a984-3db4-4e54-93da-f857384f5316" (UID: "c5e6a984-3db4-4e54-93da-f857384f5316"). InnerVolumeSpecName "kube-api-access-kkgx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:01:56.851040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.850955 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgx7\" (UniqueName: \"kubernetes.io/projected/c5e6a984-3db4-4e54-93da-f857384f5316-kube-api-access-kkgx7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:56.851040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:56.850985 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfxh\" (UniqueName: \"kubernetes.io/projected/732e98ae-cc72-48a1-b61f-c31f53b32bd3-kube-api-access-tzfxh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:01:57.524905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.524859 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:57.643159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.643069 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" Nov 02 23:01:57.643159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.643066 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk" event={"ID":"732e98ae-cc72-48a1-b61f-c31f53b32bd3","Type":"ContainerDied","Data":"e25d381a4b4eed5c712d69f8534bcb17b5373ccd56979d9de8d2a50ac640af42"} Nov 02 23:01:57.643426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.643267 4494 scope.go:122] "RemoveContainer" containerID="92e37d09c51b284fed5cbbbf469630a372d3f690230bc31ef06ae36c9cf45fad" Nov 02 23:01:57.644782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.644515 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" event={"ID":"c5e6a984-3db4-4e54-93da-f857384f5316","Type":"ContainerDied","Data":"13c23cf68448238080c5ee02779af852902448bb7368669c0b5eff597aba6b0f"} Nov 02 23:01:57.644782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.644553 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg" Nov 02 23:01:57.653061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.653045 4494 scope.go:122] "RemoveContainer" containerID="e971df1cfb54db9e31390c366b361ca18a34a37f8208e447a0ad3dc544226d96" Nov 02 23:01:57.812549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.812515 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:01:57.816201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.816165 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-708f46a0-7f69ff7fbc-vl8pg"] Nov 02 23:01:57.927295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.927263 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:01:57.931238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:57.931213 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-708f46a0-78dd4c758c-58dmk"] Nov 02 23:01:58.594429 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:58.594394 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:58.750540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:58.750509 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:58.895874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:58.895769 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:59.692579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:59.692547 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" path="/var/lib/kubelet/pods/732e98ae-cc72-48a1-b61f-c31f53b32bd3/volumes" Nov 02 23:01:59.692989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:59.692906 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e6a984-3db4-4e54-93da-f857384f5316" path="/var/lib/kubelet/pods/c5e6a984-3db4-4e54-93da-f857384f5316/volumes" Nov 02 23:01:59.734112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:01:59.734073 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:00.148498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148420 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 02 23:02:00.148835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148824 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c5e6a984-3db4-4e54-93da-f857384f5316" containerName="payload-pre-processing" Nov 02 23:02:00.148882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148838 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e6a984-3db4-4e54-93da-f857384f5316" containerName="payload-pre-processing" Nov 02 23:02:00.148882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148875 4494 cpu_manager.go:414] "containerMap: removing container" podUID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" containerName="payload-processing" Nov 02 23:02:00.148882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148880 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" containerName="payload-processing" Nov 02 23:02:00.148973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148890 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.148973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148895 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.148973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148903 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.148973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148909 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.148973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148971 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="732e98ae-cc72-48a1-b61f-c31f53b32bd3" containerName="payload-processing" Nov 02 23:02:00.149114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148978 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.149114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148984 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be0d79e-06f2-40ce-98d2-e8514a0c8895" containerName="istio-proxy" Nov 02 23:02:00.149114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.148993 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e6a984-3db4-4e54-93da-f857384f5316" containerName="payload-pre-processing" Nov 02 23:02:00.153256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.153241 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.182385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.182361 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.182540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.182509 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5rc\" (UniqueName: \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.211585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.211547 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-637fdb\"" Nov 02 23:02:00.219635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.219597 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 02 23:02:00.262366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.262339 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-637fdb-dockercfg-wdgnx\"" Nov 02 23:02:00.284137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.284101 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.284360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.284284 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5rc\" (UniqueName: \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.284936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.284918 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.314962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.314932 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 02 23:02:00.318648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.318633 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.349497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.349467 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5rc\" (UniqueName: \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\") pod \"payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.357163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.357136 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 02 23:02:00.384915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.384881 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjsh\" (UniqueName: \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.385105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.384976 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.466716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.466686 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:00.485813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.485780 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjsh\" (UniqueName: \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.486330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.486305 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.487207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.487149 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.547728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.547702 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjsh\" (UniqueName: \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\") pod \"payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.599358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.599329 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 02 23:02:00.631346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.631309 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:00.846487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:00.846463 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 02 23:02:00.848024 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:02:00.847999 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726e5179_b6a5_48d5_a8c3_a570d90e97f2.slice/crio-58cf7040f7aa397e20c1e8ccbedd9cf9028db8e23b2eb62d920a564a13710ecd WatchSource:0}: Error finding container 58cf7040f7aa397e20c1e8ccbedd9cf9028db8e23b2eb62d920a564a13710ecd: Status 404 returned error can't find the container with id 58cf7040f7aa397e20c1e8ccbedd9cf9028db8e23b2eb62d920a564a13710ecd Nov 02 23:02:01.004770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.004746 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 02 23:02:01.006445 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:02:01.006402 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2259f9bc_a646_45b1_9ba4_71ae5038d540.slice/crio-93c26a243945c486270566b0d27d49e1163b0948a7f1a0dfa45bdafba93df99b WatchSource:0}: Error finding container 93c26a243945c486270566b0d27d49e1163b0948a7f1a0dfa45bdafba93df99b: Status 404 returned error can't find the container with id 93c26a243945c486270566b0d27d49e1163b0948a7f1a0dfa45bdafba93df99b Nov 02 23:02:01.029198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.029157 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:01.691981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.691950 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:01.692160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.691999 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:01.692160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.692015 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" event={"ID":"726e5179-b6a5-48d5-a8c3-a570d90e97f2","Type":"ContainerStarted","Data":"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578"} Nov 02 23:02:01.692160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.692033 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" event={"ID":"726e5179-b6a5-48d5-a8c3-a570d90e97f2","Type":"ContainerStarted","Data":"58cf7040f7aa397e20c1e8ccbedd9cf9028db8e23b2eb62d920a564a13710ecd"} Nov 02 23:02:01.692160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.692048 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" event={"ID":"2259f9bc-a646-45b1-9ba4-71ae5038d540","Type":"ContainerStarted","Data":"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68"} Nov 02 23:02:01.692160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.692062 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" event={"ID":"2259f9bc-a646-45b1-9ba4-71ae5038d540","Type":"ContainerStarted","Data":"93c26a243945c486270566b0d27d49e1163b0948a7f1a0dfa45bdafba93df99b"} Nov 02 23:02:01.803693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.803640 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" podStartSLOduration=1.803625425 podStartE2EDuration="1.803625425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:01.802334466 +0000 UTC m=+836.232204338" watchObservedRunningTime="2026-11-02 23:02:01.803625425 +0000 UTC m=+836.233495287" Nov 02 23:02:01.921091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:01.921024 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" podStartSLOduration=1.9210004729999999 podStartE2EDuration="1.921000473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:01.918665298 +0000 UTC m=+836.348535160" watchObservedRunningTime="2026-11-02 23:02:01.921000473 +0000 UTC m=+836.350870336" Nov 02 23:02:03.074986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:03.074951 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:04.594187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:04.594144 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:04.873788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:04.873703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:05.018094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:05.018062 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:05.092379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:05.092337 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7"] Nov 02 23:02:05.143338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:05.143261 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-708f46a0-vwxv7"] Nov 02 23:02:05.206491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:05.206451 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:05.692828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:05.692800 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e75b5a-d2c5-4741-8270-6a982a0f6e87" path="/var/lib/kubelet/pods/39e75b5a-d2c5-4741-8270-6a982a0f6e87/volumes" Nov 02 23:02:06.043816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.043705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:06.073793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.073765 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:02:06.556044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.556002 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:02:06.562244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.562213 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.627135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.627104 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42983-openshift-default-dockercfg-xchbk\"" Nov 02 23:02:06.658223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.658048 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:06.661653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661612 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661660 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661677 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661698 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661744 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661776 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.661799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661796 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.662162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661813 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrt59\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.662162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.661855 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.679761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.679729 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:02:06.762999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.762965 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrt59\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763033 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763066 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763100 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763116 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763137 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763274 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763554 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763676 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763790 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.763842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763824 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.764035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.763864 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.764748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.764719 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.765808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.765783 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.766147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.766125 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.805591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.805495 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:06.824902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.824832 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.824902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.824887 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrt59\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:06.875972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:06.875936 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:07.326402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.326373 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:02:07.328373 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:02:07.328340 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2107d46e_3bab_48f9_924a_ba2d841bfd4a.slice/crio-cde00341d658c366e677c3b8a178ef2e26ae967a729bd40de5deeaa3a1d0ad8f WatchSource:0}: Error finding container cde00341d658c366e677c3b8a178ef2e26ae967a729bd40de5deeaa3a1d0ad8f: Status 404 returned error can't find the container with id cde00341d658c366e677c3b8a178ef2e26ae967a729bd40de5deeaa3a1d0ad8f Nov 02 23:02:07.341195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.330904 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:07.341195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.330993 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:07.341195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.331037 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:07.724696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.724658 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerStarted","Data":"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b"} Nov 02 23:02:07.724910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.724726 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerStarted","Data":"cde00341d658c366e677c3b8a178ef2e26ae967a729bd40de5deeaa3a1d0ad8f"} Nov 02 23:02:07.855312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.847429 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podStartSLOduration=1.847409619 podStartE2EDuration="1.847409619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:07.846250294 +0000 UTC m=+842.276120235" watchObservedRunningTime="2026-11-02 23:02:07.847409619 +0000 UTC m=+842.277279483" Nov 02 23:02:07.876584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:07.876553 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:08.882022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:08.881982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:09.758324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:09.758285 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:09.759297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:09.759278 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:10.214054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:10.214014 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:10.725145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:10.725108 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:10.895606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:10.895569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:11.105795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:11.105679 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:12.696333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:12.696306 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 02 23:02:12.696794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:12.696368 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 02 23:02:13.928128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:13.928057 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:14.104214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.104156 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:14.348082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.347994 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 02 23:02:14.353191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.353151 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.416934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.416902 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 02 23:02:14.456274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.456241 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8fg\" (UniqueName: \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.456438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.456405 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.463241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.463219 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b42983\"" Nov 02 23:02:14.557509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.557457 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8fg\" (UniqueName: \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.557950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.557762 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.558660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.558633 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.620473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.620376 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8fg\" (UniqueName: \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\") pod \"payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:14.666477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:14.666434 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:15.049883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.049852 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 02 23:02:15.051690 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:02:15.051661 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f343c1_f72a_4950_9434_5a3dcd238c1d.slice/crio-fdbd04d08645a8924d6c6591a45134bcf4098e3af40303a9a3e1c2d935f34381 WatchSource:0}: Error finding container fdbd04d08645a8924d6c6591a45134bcf4098e3af40303a9a3e1c2d935f34381: Status 404 returned error can't find the container with id fdbd04d08645a8924d6c6591a45134bcf4098e3af40303a9a3e1c2d935f34381 Nov 02 23:02:15.413285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.413259 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:15.570181 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.570134 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:15.714622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.714540 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:15.793317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.793285 4494 generic.go:356] "Generic (PLEG): container finished" podID="61a87125-f959-4252-8e96-ede7e8de6a14" containerID="f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948" exitCode=0 Nov 02 23:02:15.793486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.793356 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerDied","Data":"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948"} Nov 02 23:02:15.793782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.793765 4494 scope.go:122] "RemoveContainer" containerID="f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948" Nov 02 23:02:15.795084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795055 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:15.795201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795085 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" event={"ID":"44f343c1-f72a-4950-9434-5a3dcd238c1d","Type":"ContainerStarted","Data":"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b"} Nov 02 23:02:15.795201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795131 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" event={"ID":"44f343c1-f72a-4950-9434-5a3dcd238c1d","Type":"ContainerStarted","Data":"fdbd04d08645a8924d6c6591a45134bcf4098e3af40303a9a3e1c2d935f34381"} Nov 02 23:02:15.795201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795136 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:15.795201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795156 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:15.795201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:15.795165 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:16.084335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:16.084228 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" podStartSLOduration=2.084212613 podStartE2EDuration="2.084212613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:16.081971569 +0000 UTC m=+850.511841424" watchObservedRunningTime="2026-11-02 23:02:16.084212613 +0000 UTC m=+850.514082482" Nov 02 23:02:16.802106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:16.802069 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerStarted","Data":"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283"} Nov 02 23:02:17.592379 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:02:17.592281 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:02:17.648959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:17.648928 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:02:17.650327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:17.650301 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 23:02:17.650550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:17.650522 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 23:02:18.652313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:18.652270 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 23:02:18.652926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:18.652890 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 23:02:19.476191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:19.476145 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:02:19.481528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:19.481491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:19.646686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:19.646648 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 23:02:19.646978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:19.646950 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 23:02:20.087685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:20.087649 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:20.254625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:20.254590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:20.641518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:20.641481 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 23:02:20.641814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:20.641786 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 23:02:21.082524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:21.082492 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:21.646636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:21.646598 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 23:02:21.647292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:21.647257 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 23:02:22.643675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:22.643642 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:02:22.644902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:22.644875 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 02 23:02:23.798082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:23.798047 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:24.075305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:24.075195 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:02:26.280414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:26.280376 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:26.805519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:26.805493 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 02 23:02:31.076030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:31.075991 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:31.578726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:31.578685 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:31.728498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:31.728461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:31.874762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:31.874691 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:02:31.891537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:31.891502 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:33.578646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.578570 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:33.578646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.578619 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:33.906843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.906806 4494 generic.go:356] "Generic (PLEG): container finished" podID="4625b18a-a96b-4208-b22d-cc49548112a0" containerID="73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d" exitCode=0 Nov 02 23:02:33.907025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.906893 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerDied","Data":"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d"} Nov 02 23:02:33.907337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.907308 4494 scope.go:122] "RemoveContainer" containerID="73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d" Nov 02 23:02:33.908614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.908593 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:33.908670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.908654 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:33.908708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:33.908687 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:34.914298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:34.914262 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerStarted","Data":"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5"} Nov 02 23:02:35.521472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:35.521445 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:02:35.521674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:35.521655 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:35.521732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:35.521712 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:36.522122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:36.522086 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:36.522550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:36.522138 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:36.722252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:36.722198 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:36.918208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:36.918154 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:37.101474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:37.101428 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:37.328385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:37.328301 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:37.481892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:37.481863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:37.521437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:37.521401 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:37.521597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:37.521473 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:38.521752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:38.521715 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:38.522153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:38.521769 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:39.521526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:39.521486 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 02 23:02:39.521715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:39.521541 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 02 23:02:40.525750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:40.525723 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:02:40.526217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:40.526152 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:02:40.526940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:40.526921 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:02:42.451340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:42.451308 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:46.253828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:46.253794 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:02:47.485162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:47.485111 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:47.606133 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:02:47.606095 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:02:48.017364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:48.017327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:52.060949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.059759 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:02:52.064515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.064494 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172293 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172349 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172456 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172512 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172613 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172676 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c682r\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172721 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172790 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.172872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.172855 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.185856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.185821 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:02:52.273926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.273898 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.273933 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.273981 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274034 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274195 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274261 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274298 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274329 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274378 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c682r\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274407 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.274508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.274447 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.275125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.275102 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.275442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.275418 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.275539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.275477 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.277465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.277442 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.277743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.277723 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.335970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.335890 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c682r\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.335970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.335900 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.377776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.377744 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:52.581838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.581792 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:52.813878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.813832 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:02:52.814188 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:02:52.814146 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09424ab0_6c4b_4b2f_9369_0663f13c3ac4.slice/crio-d6e1a2b51b3d79527f72398f946a69dc61a8eef0781ab6effec054970681ef9f WatchSource:0}: Error finding container d6e1a2b51b3d79527f72398f946a69dc61a8eef0781ab6effec054970681ef9f: Status 404 returned error can't find the container with id d6e1a2b51b3d79527f72398f946a69dc61a8eef0781ab6effec054970681ef9f Nov 02 23:02:52.816630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.816596 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:52.816722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.816679 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:52.816801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:52.816723 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:53.048796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.048728 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" event={"ID":"09424ab0-6c4b-4b2f-9369-0663f13c3ac4","Type":"ContainerStarted","Data":"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563"} Nov 02 23:02:53.051393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.048851 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" event={"ID":"09424ab0-6c4b-4b2f-9369-0663f13c3ac4","Type":"ContainerStarted","Data":"d6e1a2b51b3d79527f72398f946a69dc61a8eef0781ab6effec054970681ef9f"} Nov 02 23:02:53.171541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.171483 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podStartSLOduration=1.171468248 podStartE2EDuration="1.171468248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:53.170151192 +0000 UTC m=+887.600021090" watchObservedRunningTime="2026-11-02 23:02:53.171468248 +0000 UTC m=+887.601338158" Nov 02 23:02:53.378348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.378317 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:53.380343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.380318 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:53.380494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:53.380366 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:54.378997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:54.378954 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:54.379532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:54.379009 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:55.150198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:55.130945 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:02:55.379047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:55.378695 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:55.379047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:55.378742 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:55.802370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:55.802333 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:02:55.802568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:55.802402 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:02:56.378631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:56.378594 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:56.378850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:56.378651 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:56.599887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:56.599764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:57.079412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.079377 4494 generic.go:356] "Generic (PLEG): container finished" podID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerID="1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b" exitCode=0 Nov 02 23:02:57.079591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.079448 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerDied","Data":"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b"} Nov 02 23:02:57.079841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.079825 4494 scope.go:122] "RemoveContainer" containerID="1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b" Nov 02 23:02:57.081235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.081208 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:57.081292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.081277 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:57.081327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.081314 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:02:57.198927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.198894 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:57.369110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.368939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:57.379166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.379135 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:57.379335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.379209 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:57.560151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:57.555996 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:58.091106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.091071 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerStarted","Data":"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c"} Nov 02 23:02:58.115884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.115855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:58.378737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.378631 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 23:02:58.379030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.378990 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 23:02:58.876443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.876392 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:02:58.877278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.877241 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:02:58.877509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:58.877462 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 23:02:59.382761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:59.382728 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:59.383240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:59.383007 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:59.383850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:59.383828 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:02:59.876868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:59.876832 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:02:59.877053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:02:59.876899 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 23:03:00.222898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:00.222864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:00.877133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:00.877097 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:03:00.877628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:00.877164 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 23:03:01.876787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:01.876748 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:03:01.876970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:01.876808 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 23:03:02.876917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:02.876881 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 23:03:02.877436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:02.876954 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 23:03:03.190036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:03.190007 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:03.880394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:03.880363 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:03:03.880786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:03.880691 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:03:03.881381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:03.881361 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:03:08.161751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:08.161718 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:11.207161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:11.207129 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:03:11.676493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:11.676156 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:13.057482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:13.057449 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:16.684132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:16.684064 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:17.311901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:17.311863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:17.458356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:17.458324 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:17.567748 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:03:17.567659 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:03:17.607759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:17.607728 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:18.173321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:18.173289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:21.717638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:21.717600 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:22.283667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.283630 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 02 23:03:22.287464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.287439 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.388980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.388948 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.388980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.388979 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389003 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389211 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389278 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389396 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389475 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389506 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6klg\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.389605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.389577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490241 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490291 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6klg\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490347 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490401 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490424 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490457 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490530 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490562 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490640 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490699 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.490919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490788 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 02 23:03:22.491640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.490994 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.491640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.491269 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.491766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.491740 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.491859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.491836 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.494236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.494189 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.494435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.494417 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.554627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.554537 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6klg\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.555009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.554987 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:22.601721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:22.601692 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:23.031822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.031780 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 02 23:03:23.068835 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:03:23.068808 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07b85b5_7585_40ac_bd0a_feeb294cafa5.slice/crio-bf91a9ad48ba84270031aed90dc868af6974aec3b9d123ce3a6b5fef6ba7c1e9 WatchSource:0}: Error finding container bf91a9ad48ba84270031aed90dc868af6974aec3b9d123ce3a6b5fef6ba7c1e9: Status 404 returned error can't find the container with id bf91a9ad48ba84270031aed90dc868af6974aec3b9d123ce3a6b5fef6ba7c1e9 Nov 02 23:03:23.071264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.071234 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.071361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.071298 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.071361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.071326 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:03:23.153271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.153232 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:23.238623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.238589 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" event={"ID":"d07b85b5-7585-40ac-bd0a-feeb294cafa5","Type":"ContainerStarted","Data":"a33026425f628997739b68f9d7bc6180d867e709fdfaff3556ec585d02336796"} Nov 02 23:03:23.238807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.238654 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" event={"ID":"d07b85b5-7585-40ac-bd0a-feeb294cafa5","Type":"ContainerStarted","Data":"bf91a9ad48ba84270031aed90dc868af6974aec3b9d123ce3a6b5fef6ba7c1e9"} Nov 02 23:03:23.361860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.361769 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podStartSLOduration=1.361750142 podStartE2EDuration="1.361750142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.35993084 +0000 UTC m=+917.789800709" watchObservedRunningTime="2026-11-02 23:03:23.361750142 +0000 UTC m=+917.791620002" Nov 02 23:03:23.602348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:23.602317 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:24.603313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:24.603277 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:03:24.603724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:24.603326 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 23:03:25.602542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:25.602505 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:03:25.602719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:25.602559 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 23:03:26.602044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:26.602009 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:03:26.602547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:26.602071 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 23:03:27.602766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:27.602734 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:03:27.603204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:27.602793 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 23:03:27.925693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:27.925663 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:03:28.183360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:28.183270 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:28.328710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:28.328674 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:28.602478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:28.602392 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:03:28.602478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:28.602452 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 23:03:29.607139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:29.607109 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:29.607569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:29.607477 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:29.608353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:29.608337 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 02 23:03:33.064069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:33.064035 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:38.049935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:38.049890 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:39.347546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:39.347508 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:43.059767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:43.059727 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:47.559258 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:03:47.559161 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:03:47.950486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:47.950453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:52.864969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:52.864929 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:53.451503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:53.451464 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:03:53.490111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:53.490079 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:57.856907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:03:57.856870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:00.626654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:00.626567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:02.760645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.760613 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:02.948445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.948410 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 02 23:04:02.952473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.952450 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988217 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988264 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988400 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988480 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988571 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j55t\" (UniqueName: \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:02.988645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:02.988623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.010067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.010037 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 02 23:04:03.034820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.034749 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:03.058129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.058103 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"default-dockercfg-smtpz\"" Nov 02 23:04:03.058129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.058101 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"openshift-service-ca.crt\"" Nov 02 23:04:03.058321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.058151 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"premium-w3-96031a-kserve-self-signed-certs\"" Nov 02 23:04:03.058321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.058154 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"kube-root-ca.crt\"" Nov 02 23:04:03.100720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.100605 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.101159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.101123 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j55t\" (UniqueName: \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.102115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.102093 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.102680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.102657 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.102850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.102833 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.103086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.103056 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.103927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.101686 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.104398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.102594 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.104588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.104341 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.113919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.113869 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.114102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.113934 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.163642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.163599 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j55t\" (UniqueName: \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\") pod \"premium-w3-96031a-kserve-fb668585f-hmdxc\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.266394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.266360 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:03.661752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:03.661723 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 02 23:04:03.663471 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:03.663441 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab4b6c4_d139_4420_993c_120f78771e0b.slice/crio-c170d7fc2bf8a1e48db2b5a6676c755641303996988617aabee2374635dcfddc WatchSource:0}: Error finding container c170d7fc2bf8a1e48db2b5a6676c755641303996988617aabee2374635dcfddc: Status 404 returned error can't find the container with id c170d7fc2bf8a1e48db2b5a6676c755641303996988617aabee2374635dcfddc Nov 02 23:04:04.480990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:04.480954 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" event={"ID":"fab4b6c4-d139-4420-993c-120f78771e0b","Type":"ContainerStarted","Data":"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58"} Nov 02 23:04:04.481424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:04.481000 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" event={"ID":"fab4b6c4-d139-4420-993c-120f78771e0b","Type":"ContainerStarted","Data":"c170d7fc2bf8a1e48db2b5a6676c755641303996988617aabee2374635dcfddc"} Nov 02 23:04:04.602682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:04.602627 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podStartSLOduration=2.602615523 podStartE2EDuration="2.602615523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:04.601802821 +0000 UTC m=+959.031672682" watchObservedRunningTime="2026-11-02 23:04:04.602615523 +0000 UTC m=+959.032485384" Nov 02 23:04:05.266656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:05.266627 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:07.746425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:07.746393 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:09.546773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.546739 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b"] Nov 02 23:04:09.550779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.550759 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.634641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.634603 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:04:09.634831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.634711 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:04:09.636988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.636954 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b"] Nov 02 23:04:09.687102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.687066 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.687329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.687236 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcp9z\" (UniqueName: \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.687329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.687275 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.704844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.704818 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:04:09.788903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.788870 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp9z\" (UniqueName: \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.789090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.788934 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.789297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.789258 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.789873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.789850 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.792037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.792018 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:09.881791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:09.881705 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp9z\" (UniqueName: \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\") pod \"maas-api-revoke-sub-19a94d0f-md56b\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:10.164328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:10.164277 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:10.597688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:10.597657 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b"] Nov 02 23:04:10.598859 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:10.598837 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5e760d_f79a_470e_affc_f590970f9fa5.slice/crio-b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc WatchSource:0}: Error finding container b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc: Status 404 returned error can't find the container with id b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc Nov 02 23:04:11.523064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:11.523026 4494 generic.go:356] "Generic (PLEG): container finished" podID="ed5e760d-f79a-470e-affc-f590970f9fa5" containerID="79ea11144c219de04130d57b37a7b62504e1f12b08734e534402780af69a633c" exitCode=0 Nov 02 23:04:11.523244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:11.523090 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" event={"ID":"ed5e760d-f79a-470e-affc-f590970f9fa5","Type":"ContainerDied","Data":"79ea11144c219de04130d57b37a7b62504e1f12b08734e534402780af69a633c"} Nov 02 23:04:11.523244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:11.523143 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" event={"ID":"ed5e760d-f79a-470e-affc-f590970f9fa5","Type":"ContainerStarted","Data":"b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc"} Nov 02 23:04:12.664880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.664852 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:12.670014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.669996 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:12.838868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.838786 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\") pod \"ed5e760d-f79a-470e-affc-f590970f9fa5\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " Nov 02 23:04:12.838868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.838844 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\") pod \"ed5e760d-f79a-470e-affc-f590970f9fa5\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " Nov 02 23:04:12.839070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.838944 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\" (UniqueName: \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\") pod \"ed5e760d-f79a-470e-affc-f590970f9fa5\" (UID: \"ed5e760d-f79a-470e-affc-f590970f9fa5\") " Nov 02 23:04:12.839317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.839294 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca" pod "ed5e760d-f79a-470e-affc-f590970f9fa5" (UID: "ed5e760d-f79a-470e-affc-f590970f9fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:12.839599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.839560 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:12.841116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.841092 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca" pod "ed5e760d-f79a-470e-affc-f590970f9fa5" (UID: "ed5e760d-f79a-470e-affc-f590970f9fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:04:12.841226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.841136 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z" pod "ed5e760d-f79a-470e-affc-f590970f9fa5" (UID: "ed5e760d-f79a-470e-affc-f590970f9fa5"). InnerVolumeSpecName "kube-api-access-kcp9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:12.940422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.940393 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5e760d-f79a-470e-affc-f590970f9fa5-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:12.940422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:12.940419 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcp9z\" (UniqueName: \"kubernetes.io/projected/ed5e760d-f79a-470e-affc-f590970f9fa5-kube-api-access-kcp9z\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:13.267016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.266982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:13.279350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.279329 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:13.535478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.535374 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" Nov 02 23:04:13.535478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.535426 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b" event={"ID":"ed5e760d-f79a-470e-affc-f590970f9fa5","Type":"ContainerDied","Data":"b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc"} Nov 02 23:04:13.535478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.535475 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b757ea8e91674d7e2b2bb6e58dc1e3130fff4735bb369e7da02baafb4331cbcc" Nov 02 23:04:13.553405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:13.553378 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 02 23:04:15.885794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.885756 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:04:15.886358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.886343 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ed5e760d-f79a-470e-affc-f590970f9fa5" containerName="revoke-keys" Nov 02 23:04:15.886420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.886360 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5e760d-f79a-470e-affc-f590970f9fa5" containerName="revoke-keys" Nov 02 23:04:15.886465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.886434 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5e760d-f79a-470e-affc-f590970f9fa5" containerName="revoke-keys" Nov 02 23:04:15.889638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.889619 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.943084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.943056 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-08c2c6-openshift-default-dockercfg-qcc5b\"" Nov 02 23:04:15.972686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.972656 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.972848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.972693 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.972848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.972793 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxqf\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.972928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.972904 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.972964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.972946 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.973062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.973046 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.973124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.973082 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.973164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.973132 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.973164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.973155 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:15.996724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:15.996700 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:04:16.074453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074424 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074462 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074513 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxqf\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074579 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074614 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074661 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074791 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074875 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.074931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074913 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.075058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074929 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.075058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.075023 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.075058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.074908 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.075347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.075326 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.076225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.076201 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.077305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.077276 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.077477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.077456 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.136192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.136104 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxqf\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.136339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.136215 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.204193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.204138 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:16.631495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.631472 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:04:16.633295 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:16.633264 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e806f4d_804a_46b3_91ef_176211c3841d.slice/crio-d556ae2f622c2e5e72d5c190c65e78c7574e1e0739b8b74141ad76d29c7db537 WatchSource:0}: Error finding container d556ae2f622c2e5e72d5c190c65e78c7574e1e0739b8b74141ad76d29c7db537: Status 404 returned error can't find the container with id d556ae2f622c2e5e72d5c190c65e78c7574e1e0739b8b74141ad76d29c7db537 Nov 02 23:04:16.635550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.635520 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:04:16.635625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.635611 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:04:16.635673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:16.635638 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:04:17.569213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:17.567640 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:17.573191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:17.571863 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" event={"ID":"9e806f4d-804a-46b3-91ef-176211c3841d","Type":"ContainerStarted","Data":"4a3133904d0f46278e080e806837121598cca1ab64b70371b9349789f4cba966"} Nov 02 23:04:17.573191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:17.571927 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" event={"ID":"9e806f4d-804a-46b3-91ef-176211c3841d","Type":"ContainerStarted","Data":"d556ae2f622c2e5e72d5c190c65e78c7574e1e0739b8b74141ad76d29c7db537"} Nov 02 23:04:17.587788 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:04:17.587758 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:04:17.799772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:17.799722 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podStartSLOduration=2.799706987 podStartE2EDuration="2.799706987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:17.748796573 +0000 UTC m=+972.178666478" watchObservedRunningTime="2026-11-02 23:04:17.799706987 +0000 UTC m=+972.229576848" Nov 02 23:04:18.204843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:18.204808 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:18.206515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:18.206492 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:18.206595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:18.206543 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:19.204717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:19.204678 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:19.205213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:19.204743 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:20.204524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:20.204487 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:20.204700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:20.204540 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:21.205538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:21.205501 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:21.206025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:21.205576 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:22.204627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:22.204590 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:22.204865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:22.204653 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:22.579377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:22.579287 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:23.205569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:23.205529 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:04:23.205764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:23.205599 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 23:04:25.090332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:25.090290 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:25.090853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:25.090710 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:25.091517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:25.091501 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:04:27.488463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.488430 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld"] Nov 02 23:04:27.492243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.492220 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.546913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.546880 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:04:27.552658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.552630 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:27.598524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.598492 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:04:27.602451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.602417 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld"] Nov 02 23:04:27.634568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.633760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.634568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.633812 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.634568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.633854 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tf6\" (UniqueName: \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.734774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.734732 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.734774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.734777 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.735066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.734893 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tf6\" (UniqueName: \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.735520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.735494 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.737277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.737254 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.797767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.797680 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tf6\" (UniqueName: \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\") pod \"maas-api-revoke-sub-fb3feda2-wpfld\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:27.806364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:27.806338 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:28.179523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:28.179496 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld"] Nov 02 23:04:28.181370 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:28.181344 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef9bb5c_a046_4106_9a53_4e0aceb2ce84.slice/crio-e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d WatchSource:0}: Error finding container e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d: Status 404 returned error can't find the container with id e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d Nov 02 23:04:28.643843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:28.643799 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" event={"ID":"fef9bb5c-a046-4106-9a53-4e0aceb2ce84","Type":"ContainerStarted","Data":"aa733f0de8d8231b3a67983de3ec73dc79e3062effc106914943a5b0a2e93e8c"} Nov 02 23:04:28.644306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:28.643859 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" event={"ID":"fef9bb5c-a046-4106-9a53-4e0aceb2ce84","Type":"ContainerStarted","Data":"e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d"} Nov 02 23:04:28.761887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:28.761776 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" podStartSLOduration=1.761762871 podStartE2EDuration="1.761762871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:28.76051639 +0000 UTC m=+983.190386252" watchObservedRunningTime="2026-11-02 23:04:28.761762871 +0000 UTC m=+983.191632732" Nov 02 23:04:29.650627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:29.650587 4494 generic.go:356] "Generic (PLEG): container finished" podID="fef9bb5c-a046-4106-9a53-4e0aceb2ce84" containerID="aa733f0de8d8231b3a67983de3ec73dc79e3062effc106914943a5b0a2e93e8c" exitCode=0 Nov 02 23:04:29.651063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:29.650674 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" event={"ID":"fef9bb5c-a046-4106-9a53-4e0aceb2ce84","Type":"ContainerDied","Data":"aa733f0de8d8231b3a67983de3ec73dc79e3062effc106914943a5b0a2e93e8c"} Nov 02 23:04:30.786302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.786272 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:30.876954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.876923 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\" (UniqueName: \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\") pod \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " Nov 02 23:04:30.877129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.876975 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\") pod \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " Nov 02 23:04:30.877129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.877106 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\") pod \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\" (UID: \"fef9bb5c-a046-4106-9a53-4e0aceb2ce84\") " Nov 02 23:04:30.877494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.877463 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca" pod "fef9bb5c-a046-4106-9a53-4e0aceb2ce84" (UID: "fef9bb5c-a046-4106-9a53-4e0aceb2ce84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:30.879110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.879087 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6" pod "fef9bb5c-a046-4106-9a53-4e0aceb2ce84" (UID: "fef9bb5c-a046-4106-9a53-4e0aceb2ce84"). InnerVolumeSpecName "kube-api-access-m6tf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:30.879789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.879768 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca" pod "fef9bb5c-a046-4106-9a53-4e0aceb2ce84" (UID: "fef9bb5c-a046-4106-9a53-4e0aceb2ce84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:04:30.934583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.934523 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:30.984324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.979034 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6tf6\" (UniqueName: \"kubernetes.io/projected/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-kube-api-access-m6tf6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:30.984324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.979075 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:30.984324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:30.979092 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef9bb5c-a046-4106-9a53-4e0aceb2ce84-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:04:31.664987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:31.664961 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" Nov 02 23:04:31.665211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:31.664962 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld" event={"ID":"fef9bb5c-a046-4106-9a53-4e0aceb2ce84","Type":"ContainerDied","Data":"e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d"} Nov 02 23:04:31.665211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:31.665089 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ac15ca0096b2fcb3517911c9acbc876c6fb0556b223b41335f93a71310123d" Nov 02 23:04:32.449793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:32.449758 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:37.450186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:37.450105 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:38.140647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:38.140604 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:04:41.735387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:41.735352 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:41.880315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:41.880279 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:42.428480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:42.428448 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:47.341144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:47.341110 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:47.561607 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:04:47.561572 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:04:52.253444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:52.253395 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:55.339583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:55.339537 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:56.966080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:56.966045 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77"] Nov 02 23:04:56.966608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:56.966592 4494 cpu_manager.go:414] "containerMap: removing container" podUID="fef9bb5c-a046-4106-9a53-4e0aceb2ce84" containerName="revoke-keys" Nov 02 23:04:56.966656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:56.966610 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef9bb5c-a046-4106-9a53-4e0aceb2ce84" containerName="revoke-keys" Nov 02 23:04:56.966711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:56.966701 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef9bb5c-a046-4106-9a53-4e0aceb2ce84" containerName="revoke-keys" Nov 02 23:04:56.970165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:56.970149 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.014601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.014566 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wggm\" (UniqueName: \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.014765 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.014621 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.014806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.014779 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.026148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.026108 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77"] Nov 02 23:04:57.074268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.074235 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:04:57.074436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.074279 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:04:57.116075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.116040 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wggm\" (UniqueName: \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.116283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.116089 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.116283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.116158 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.116817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.116798 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.118595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.118575 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.162090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.162062 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:57.178096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.178067 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wggm\" (UniqueName: \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\") pod \"maas-api-revoke-sub-a837ea23-c2k77\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.282916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.282836 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:04:57.665651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.665624 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77"] Nov 02 23:04:57.667214 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:57.667154 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f51b62_6a60_4491_9d83_1ee1001d4903.slice/crio-11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68 WatchSource:0}: Error finding container 11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68: Status 404 returned error can't find the container with id 11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68 Nov 02 23:04:57.809524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:57.809490 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" event={"ID":"28f51b62-6a60-4491-9d83-1ee1001d4903","Type":"ContainerStarted","Data":"11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68"} Nov 02 23:04:58.397852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.397769 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h"] Nov 02 23:04:58.401764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.401741 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.436061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.436018 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5fkb\" (UniqueName: \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.436265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.436134 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.436265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.436257 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.458806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.458772 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h"] Nov 02 23:04:58.537075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.537047 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5fkb\" (UniqueName: \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.537289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.537105 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.537289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.537140 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.537980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.537944 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.540115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.540096 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.619289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.619261 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5fkb\" (UniqueName: \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\") pod \"maas-api-revoke-sub-ca737964-h7x2h\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.715954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.715918 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:04:58.818249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.818217 4494 generic.go:356] "Generic (PLEG): container finished" podID="28f51b62-6a60-4491-9d83-1ee1001d4903" containerID="4be9f8732b2b449b1179cfe656b0910b971aa9a87f3f1fc2491c9702d5ccd9c5" exitCode=0 Nov 02 23:04:58.818435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:58.818292 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" event={"ID":"28f51b62-6a60-4491-9d83-1ee1001d4903","Type":"ContainerDied","Data":"4be9f8732b2b449b1179cfe656b0910b971aa9a87f3f1fc2491c9702d5ccd9c5"} Nov 02 23:04:59.091831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:59.091798 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h"] Nov 02 23:04:59.092971 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:04:59.092942 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d96c498_7c17_4b2f_b67a_ebee25bc5696.slice/crio-feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7 WatchSource:0}: Error finding container feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7: Status 404 returned error can't find the container with id feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7 Nov 02 23:04:59.825011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:59.824972 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" event={"ID":"0d96c498-7c17-4b2f-b67a-ebee25bc5696","Type":"ContainerStarted","Data":"1e12e451885e49d4a36ec272a5cb2bb9f614279fbce96ded1e59f79d50ff78e2"} Nov 02 23:04:59.825513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:59.825041 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" event={"ID":"0d96c498-7c17-4b2f-b67a-ebee25bc5696","Type":"ContainerStarted","Data":"feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7"} Nov 02 23:04:59.943202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:59.943139 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" podStartSLOduration=1.943126726 podStartE2EDuration="1.943126726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:59.94231194 +0000 UTC m=+1014.372181808" watchObservedRunningTime="2026-11-02 23:04:59.943126726 +0000 UTC m=+1014.372996601" Nov 02 23:04:59.964762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:04:59.964735 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:05:00.053498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.053463 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\" (UniqueName: \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\") pod \"28f51b62-6a60-4491-9d83-1ee1001d4903\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " Nov 02 23:05:00.053679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.053509 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\") pod \"28f51b62-6a60-4491-9d83-1ee1001d4903\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " Nov 02 23:05:00.053679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.053586 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\") pod \"28f51b62-6a60-4491-9d83-1ee1001d4903\" (UID: \"28f51b62-6a60-4491-9d83-1ee1001d4903\") " Nov 02 23:05:00.053899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.053869 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca" pod "28f51b62-6a60-4491-9d83-1ee1001d4903" (UID: "28f51b62-6a60-4491-9d83-1ee1001d4903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:05:00.055788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.055760 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca" pod "28f51b62-6a60-4491-9d83-1ee1001d4903" (UID: "28f51b62-6a60-4491-9d83-1ee1001d4903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:05:00.055886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.055803 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm" pod "28f51b62-6a60-4491-9d83-1ee1001d4903" (UID: "28f51b62-6a60-4491-9d83-1ee1001d4903"). InnerVolumeSpecName "kube-api-access-6wggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:05:00.155515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.155415 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wggm\" (UniqueName: \"kubernetes.io/projected/28f51b62-6a60-4491-9d83-1ee1001d4903-kube-api-access-6wggm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:00.155515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.155460 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:00.155515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.155472 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f51b62-6a60-4491-9d83-1ee1001d4903-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:00.779252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.779206 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:05:00.832143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.832114 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" Nov 02 23:05:00.832628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.832149 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77" event={"ID":"28f51b62-6a60-4491-9d83-1ee1001d4903","Type":"ContainerDied","Data":"11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68"} Nov 02 23:05:00.832628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.832202 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e582d78c9c4bacd6ea833b8f1d700645879a2510ee06c03f16c188502c9a68" Nov 02 23:05:00.833804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.833781 4494 generic.go:356] "Generic (PLEG): container finished" podID="0d96c498-7c17-4b2f-b67a-ebee25bc5696" containerID="1e12e451885e49d4a36ec272a5cb2bb9f614279fbce96ded1e59f79d50ff78e2" exitCode=0 Nov 02 23:05:00.833940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.833847 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" event={"ID":"0d96c498-7c17-4b2f-b67a-ebee25bc5696","Type":"ContainerDied","Data":"1e12e451885e49d4a36ec272a5cb2bb9f614279fbce96ded1e59f79d50ff78e2"} Nov 02 23:05:00.934649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:00.934600 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:02.256499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.256474 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:05:02.329989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.329945 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:02.396308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.396270 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\") pod \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " Nov 02 23:05:02.396545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.396333 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\") pod \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " Nov 02 23:05:02.396545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.396502 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\" (UniqueName: \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\") pod \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\" (UID: \"0d96c498-7c17-4b2f-b67a-ebee25bc5696\") " Nov 02 23:05:02.398222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.397837 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca" pod "0d96c498-7c17-4b2f-b67a-ebee25bc5696" (UID: "0d96c498-7c17-4b2f-b67a-ebee25bc5696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:05:02.400693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.400632 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb" pod "0d96c498-7c17-4b2f-b67a-ebee25bc5696" (UID: "0d96c498-7c17-4b2f-b67a-ebee25bc5696"). InnerVolumeSpecName "kube-api-access-p5fkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:05:02.402208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.401375 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca" pod "0d96c498-7c17-4b2f-b67a-ebee25bc5696" (UID: "0d96c498-7c17-4b2f-b67a-ebee25bc5696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:05:02.498307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.498223 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5fkb\" (UniqueName: \"kubernetes.io/projected/0d96c498-7c17-4b2f-b67a-ebee25bc5696-kube-api-access-p5fkb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:02.498307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.498252 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:02.498307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.498263 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d96c498-7c17-4b2f-b67a-ebee25bc5696-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:02.855937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.855856 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" Nov 02 23:05:02.855937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.855877 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h" event={"ID":"0d96c498-7c17-4b2f-b67a-ebee25bc5696","Type":"ContainerDied","Data":"feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7"} Nov 02 23:05:02.855937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:02.855924 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb7201bb0a890a21d8e1169656fba1b694287cb0086e827c39c9d8d347ce6c7" Nov 02 23:05:07.262075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:07.262043 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:12.357653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.357621 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv"] Nov 02 23:05:12.358111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358097 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28f51b62-6a60-4491-9d83-1ee1001d4903" containerName="revoke-keys" Nov 02 23:05:12.358160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358111 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f51b62-6a60-4491-9d83-1ee1001d4903" containerName="revoke-keys" Nov 02 23:05:12.358160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358128 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0d96c498-7c17-4b2f-b67a-ebee25bc5696" containerName="revoke-keys" Nov 02 23:05:12.358160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358133 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d96c498-7c17-4b2f-b67a-ebee25bc5696" containerName="revoke-keys" Nov 02 23:05:12.358300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358239 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f51b62-6a60-4491-9d83-1ee1001d4903" containerName="revoke-keys" Nov 02 23:05:12.358300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.358250 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d96c498-7c17-4b2f-b67a-ebee25bc5696" containerName="revoke-keys" Nov 02 23:05:12.361688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.361651 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:12.361833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.361712 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.407433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.407404 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4549k\" (UniqueName: \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.407565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.407440 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.407565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.407458 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.420028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.419990 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv"] Nov 02 23:05:12.467040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.467007 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:05:12.467040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.467024 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:05:12.508714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.508687 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4549k\" (UniqueName: \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.508714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.508719 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.508930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.508740 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.509452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.509424 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.511207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.511186 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.570939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.570912 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4549k\" (UniqueName: \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\") pod \"maas-api-revoke-sub-bdda8137-z2qtv\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:12.675582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:12.675548 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:13.051915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:13.051892 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv"] Nov 02 23:05:13.053166 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:05:13.053143 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0936acdd_3670_43c3_8266_36446b6cd143.slice/crio-08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4 WatchSource:0}: Error finding container 08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4: Status 404 returned error can't find the container with id 08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4 Nov 02 23:05:13.926694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:13.926662 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" event={"ID":"0936acdd-3670-43c3-8266-36446b6cd143","Type":"ContainerStarted","Data":"83bac4642b795c6e3fb7f52841530f0df66c6427957eeb9e74d36cd8d1cc10d6"} Nov 02 23:05:13.927218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:13.926717 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" event={"ID":"0936acdd-3670-43c3-8266-36446b6cd143","Type":"ContainerStarted","Data":"08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4"} Nov 02 23:05:14.044054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:14.043985 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" podStartSLOduration=2.043967224 podStartE2EDuration="2.043967224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:14.043901181 +0000 UTC m=+1028.473771055" watchObservedRunningTime="2026-11-02 23:05:14.043967224 +0000 UTC m=+1028.473837091" Nov 02 23:05:14.933412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:14.933376 4494 generic.go:356] "Generic (PLEG): container finished" podID="0936acdd-3670-43c3-8266-36446b6cd143" containerID="83bac4642b795c6e3fb7f52841530f0df66c6427957eeb9e74d36cd8d1cc10d6" exitCode=0 Nov 02 23:05:14.933857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:14.933426 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" event={"ID":"0936acdd-3670-43c3-8266-36446b6cd143","Type":"ContainerDied","Data":"83bac4642b795c6e3fb7f52841530f0df66c6427957eeb9e74d36cd8d1cc10d6"} Nov 02 23:05:16.077991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.077955 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:16.151537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.151510 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\") pod \"0936acdd-3670-43c3-8266-36446b6cd143\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " Nov 02 23:05:16.151687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.151603 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\") pod \"0936acdd-3670-43c3-8266-36446b6cd143\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " Nov 02 23:05:16.151751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.151736 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\" (UniqueName: \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\") pod \"0936acdd-3670-43c3-8266-36446b6cd143\" (UID: \"0936acdd-3670-43c3-8266-36446b6cd143\") " Nov 02 23:05:16.151944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.151917 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca" pod "0936acdd-3670-43c3-8266-36446b6cd143" (UID: "0936acdd-3670-43c3-8266-36446b6cd143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:05:16.152150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.152129 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:16.153843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.153816 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca" pod "0936acdd-3670-43c3-8266-36446b6cd143" (UID: "0936acdd-3670-43c3-8266-36446b6cd143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:05:16.153998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.153982 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k" pod "0936acdd-3670-43c3-8266-36446b6cd143" (UID: "0936acdd-3670-43c3-8266-36446b6cd143"). InnerVolumeSpecName "kube-api-access-4549k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:05:16.253595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.253502 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0936acdd-3670-43c3-8266-36446b6cd143-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:16.253595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.253551 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4549k\" (UniqueName: \"kubernetes.io/projected/0936acdd-3670-43c3-8266-36446b6cd143-kube-api-access-4549k\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:16.946405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.946374 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" Nov 02 23:05:16.946405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.946394 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv" event={"ID":"0936acdd-3670-43c3-8266-36446b6cd143","Type":"ContainerDied","Data":"08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4"} Nov 02 23:05:16.946702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:16.946468 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08d9da468de57b5f8e2b4c4d4abeecdfe87670bb986a099263060dc79bde06e4" Nov 02 23:05:17.171742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:17.171687 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:17.560213 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:05:17.560107 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:05:18.930145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.930119 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr"] Nov 02 23:05:18.930661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.930646 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0936acdd-3670-43c3-8266-36446b6cd143" containerName="revoke-keys" Nov 02 23:05:18.930716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.930663 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0936acdd-3670-43c3-8266-36446b6cd143" containerName="revoke-keys" Nov 02 23:05:18.930787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.930777 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0936acdd-3670-43c3-8266-36446b6cd143" containerName="revoke-keys" Nov 02 23:05:18.934054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.934037 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:18.988263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.987877 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:05:18.996000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.993399 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:18.996000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.993467 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:18.996000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:18.993555 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqsq\" (UniqueName: \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.038283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.038250 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:05:19.040961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.040941 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr"] Nov 02 23:05:19.094665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.094634 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.094803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.094673 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.094803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.094744 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqsq\" (UniqueName: \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.095397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.095373 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.097384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.097362 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.156470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.156440 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqsq\" (UniqueName: \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\") pod \"maas-api-revoke-sub-6009e1a2-q8fzr\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.248516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.248435 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:19.626142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.626111 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr"] Nov 02 23:05:19.626434 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:05:19.626408 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cbf5f5_9071_4a3f_ac46_e4d81c284274.slice/crio-9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3 WatchSource:0}: Error finding container 9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3: Status 404 returned error can't find the container with id 9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3 Nov 02 23:05:19.964378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.964345 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" event={"ID":"78cbf5f5-9071-4a3f-ac46-e4d81c284274","Type":"ContainerStarted","Data":"a4d90a5032f3c450431b179fd25ef562fafbed4962090768882c85d524e9a504"} Nov 02 23:05:19.964804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:19.964407 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" event={"ID":"78cbf5f5-9071-4a3f-ac46-e4d81c284274","Type":"ContainerStarted","Data":"9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3"} Nov 02 23:05:20.082417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:20.082362 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" podStartSLOduration=2.082344327 podStartE2EDuration="2.082344327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:20.080496155 +0000 UTC m=+1034.510366025" watchObservedRunningTime="2026-11-02 23:05:20.082344327 +0000 UTC m=+1034.512214193" Nov 02 23:05:20.971634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:20.971598 4494 generic.go:356] "Generic (PLEG): container finished" podID="78cbf5f5-9071-4a3f-ac46-e4d81c284274" containerID="a4d90a5032f3c450431b179fd25ef562fafbed4962090768882c85d524e9a504" exitCode=0 Nov 02 23:05:20.972061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:20.971681 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" event={"ID":"78cbf5f5-9071-4a3f-ac46-e4d81c284274","Type":"ContainerDied","Data":"a4d90a5032f3c450431b179fd25ef562fafbed4962090768882c85d524e9a504"} Nov 02 23:05:22.126018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.125987 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:22.231329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.231243 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\") pod \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " Nov 02 23:05:22.231329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.231298 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\") pod \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " Nov 02 23:05:22.231531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.231426 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\" (UniqueName: \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\") pod \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\" (UID: \"78cbf5f5-9071-4a3f-ac46-e4d81c284274\") " Nov 02 23:05:22.231674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.231649 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca" pod "78cbf5f5-9071-4a3f-ac46-e4d81c284274" (UID: "78cbf5f5-9071-4a3f-ac46-e4d81c284274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:05:22.231899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.231881 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:22.233497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.233472 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca" pod "78cbf5f5-9071-4a3f-ac46-e4d81c284274" (UID: "78cbf5f5-9071-4a3f-ac46-e4d81c284274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:05:22.233636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.233615 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq" pod "78cbf5f5-9071-4a3f-ac46-e4d81c284274" (UID: "78cbf5f5-9071-4a3f-ac46-e4d81c284274"). InnerVolumeSpecName "kube-api-access-xbqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:05:22.247487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.247461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:22.333255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.333218 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbqsq\" (UniqueName: \"kubernetes.io/projected/78cbf5f5-9071-4a3f-ac46-e4d81c284274-kube-api-access-xbqsq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:22.333255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.333251 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbf5f5-9071-4a3f-ac46-e4d81c284274-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:22.985226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.985156 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" event={"ID":"78cbf5f5-9071-4a3f-ac46-e4d81c284274","Type":"ContainerDied","Data":"9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3"} Nov 02 23:05:22.985528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.985501 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9510e88695f623d66f2ea8c392f2cfff7ca82443ea23954efeb54c67f80d6ff3" Nov 02 23:05:22.985758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:22.985743 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr" Nov 02 23:05:26.994936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:26.994901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:27.141508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:27.141472 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:27.286263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:27.286133 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:29.670832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.670795 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824"] Nov 02 23:05:29.671341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.671327 4494 cpu_manager.go:414] "containerMap: removing container" podUID="78cbf5f5-9071-4a3f-ac46-e4d81c284274" containerName="revoke-keys" Nov 02 23:05:29.671382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.671343 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cbf5f5-9071-4a3f-ac46-e4d81c284274" containerName="revoke-keys" Nov 02 23:05:29.671442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.671433 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cbf5f5-9071-4a3f-ac46-e4d81c284274" containerName="revoke-keys" Nov 02 23:05:29.674611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.674592 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.728870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.728834 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:05:29.731984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.731948 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824"] Nov 02 23:05:29.779520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.779484 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:05:29.816946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.816913 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwflt\" (UniqueName: \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.817122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.816973 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.817305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.817278 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.918471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.918439 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwflt\" (UniqueName: \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.918645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.918491 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.918645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.918537 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.919138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.919117 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.921001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.920945 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.980281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.980221 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwflt\" (UniqueName: \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\") pod \"maas-api-revoke-sub-9ec36fc4-2t824\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:29.987034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:29.987013 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:30.365639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:30.365615 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824"] Nov 02 23:05:30.366135 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:05:30.366108 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac28ec94_5b4c_4565_b878_414be67b6396.slice/crio-a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2 WatchSource:0}: Error finding container a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2: Status 404 returned error can't find the container with id a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2 Nov 02 23:05:31.036808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.036776 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" event={"ID":"ac28ec94-5b4c-4565-b878-414be67b6396","Type":"ContainerStarted","Data":"cf762144c1fb77831e68c4e7dfc52b5519083f4273e9a3c5a9e24d89a16bde45"} Nov 02 23:05:31.037276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.036830 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" event={"ID":"ac28ec94-5b4c-4565-b878-414be67b6396","Type":"ContainerStarted","Data":"a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2"} Nov 02 23:05:31.156407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.156353 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" podStartSLOduration=2.156337956 podStartE2EDuration="2.156337956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:31.155923204 +0000 UTC m=+1045.585793067" watchObservedRunningTime="2026-11-02 23:05:31.156337956 +0000 UTC m=+1045.586207816" Nov 02 23:05:31.250544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.250516 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:31.760949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.760921 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:05:31.764315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.764299 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.820014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.819982 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8a8e5c7c40ee\"" Nov 02 23:05:31.823347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.823325 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:05:31.843602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.843577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.843731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.843693 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbgb\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.843813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.843798 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.843865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.843827 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.870653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.870627 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:05:31.870653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.870627 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:05:31.870836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.870631 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:05:31.870836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.870679 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:05:31.919818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.919797 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cf53d560958d\"" Nov 02 23:05:31.944438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.944414 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.944545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.944500 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbgb\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.944599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.944574 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.944636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.944603 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.945075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.945036 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.945302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.945282 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:31.947222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:31.947206 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:32.006577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.006551 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbgb\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\") pod \"payload-processing-external-model-78cb97dd58-7h4cz\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:32.044489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.044398 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:05:32.045132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.045109 4494 generic.go:356] "Generic (PLEG): container finished" podID="ac28ec94-5b4c-4565-b878-414be67b6396" containerID="cf762144c1fb77831e68c4e7dfc52b5519083f4273e9a3c5a9e24d89a16bde45" exitCode=0 Nov 02 23:05:32.045254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.045207 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" event={"ID":"ac28ec94-5b4c-4565-b878-414be67b6396","Type":"ContainerDied","Data":"cf762144c1fb77831e68c4e7dfc52b5519083f4273e9a3c5a9e24d89a16bde45"} Nov 02 23:05:32.077417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.077387 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:32.152876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.152846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:32.499594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:32.499541 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:05:32.540101 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:05:32.526603 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86713c3_09ba_40e7_b039_2617ae3f7940.slice/crio-308bdb7187316b6ff2b5a8bb6b12e565ca1340f1c87f04ffa8e5463a13691386 WatchSource:0}: Error finding container 308bdb7187316b6ff2b5a8bb6b12e565ca1340f1c87f04ffa8e5463a13691386: Status 404 returned error can't find the container with id 308bdb7187316b6ff2b5a8bb6b12e565ca1340f1c87f04ffa8e5463a13691386 Nov 02 23:05:33.083494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.083333 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" event={"ID":"c86713c3-09ba-40e7-b039-2617ae3f7940","Type":"ContainerStarted","Data":"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983"} Nov 02 23:05:33.083494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.083393 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" event={"ID":"c86713c3-09ba-40e7-b039-2617ae3f7940","Type":"ContainerStarted","Data":"308bdb7187316b6ff2b5a8bb6b12e565ca1340f1c87f04ffa8e5463a13691386"} Nov 02 23:05:33.083494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.083451 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:33.210200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.203038 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" podStartSLOduration=2.2030154299999998 podStartE2EDuration="2.20301543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:33.201609255 +0000 UTC m=+1047.631479118" watchObservedRunningTime="2026-11-02 23:05:33.20301543 +0000 UTC m=+1047.632885292" Nov 02 23:05:33.707983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.707930 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:33.802947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.802853 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\") pod \"ac28ec94-5b4c-4565-b878-414be67b6396\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " Nov 02 23:05:33.802947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.802941 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\") pod \"ac28ec94-5b4c-4565-b878-414be67b6396\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " Nov 02 23:05:33.803138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.803062 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\" (UniqueName: \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\") pod \"ac28ec94-5b4c-4565-b878-414be67b6396\" (UID: \"ac28ec94-5b4c-4565-b878-414be67b6396\") " Nov 02 23:05:33.803672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.803639 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca" pod "ac28ec94-5b4c-4565-b878-414be67b6396" (UID: "ac28ec94-5b4c-4565-b878-414be67b6396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:05:33.805838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.805804 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca" pod "ac28ec94-5b4c-4565-b878-414be67b6396" (UID: "ac28ec94-5b4c-4565-b878-414be67b6396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:05:33.806317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.806292 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt" pod "ac28ec94-5b4c-4565-b878-414be67b6396" (UID: "ac28ec94-5b4c-4565-b878-414be67b6396"). InnerVolumeSpecName "kube-api-access-fwflt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:05:33.904553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.904515 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwflt\" (UniqueName: \"kubernetes.io/projected/ac28ec94-5b4c-4565-b878-414be67b6396-kube-api-access-fwflt\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:33.904724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.904570 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:33.904724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:33.904586 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac28ec94-5b4c-4565-b878-414be67b6396-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:05:34.097498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:34.097330 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" event={"ID":"ac28ec94-5b4c-4565-b878-414be67b6396","Type":"ContainerDied","Data":"a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2"} Nov 02 23:05:34.097498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:34.097394 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2ff80da68ba2632928d9ecafadf6bd87dbab9b0f0daccab52f01f2c0c29eef2" Nov 02 23:05:34.097498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:34.097396 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824" Nov 02 23:05:37.264428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:37.264390 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:42.259846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:42.259810 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:45.108250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:45.108221 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:05:47.156687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:47.156650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:47.565743 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:05:47.565694 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:05:52.057778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:52.057739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:56.197692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.197645 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:05:56.198549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.198523 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ac28ec94-5b4c-4565-b878-414be67b6396" containerName="revoke-keys" Nov 02 23:05:56.198664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.198550 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac28ec94-5b4c-4565-b878-414be67b6396" containerName="revoke-keys" Nov 02 23:05:56.198736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.198725 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac28ec94-5b4c-4565-b878-414be67b6396" containerName="revoke-keys" Nov 02 23:05:56.202661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.202639 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.365571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.365531 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374719 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374771 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374822 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374874 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374919 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.374956 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.375029 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.375119 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.377219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.375147 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpm5\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.479696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.479595 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.482474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.482447 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.483116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.483096 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpm5\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.483698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.481641 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.483865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.483848 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.484668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.484650 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.485156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.485138 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.485657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.485637 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.487109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.487073 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.487608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.484248 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.487743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.486997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.487851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.485045 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.487987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.487968 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.488446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.488342 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.491419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.491369 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.492973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.492933 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.560397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.560313 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpm5\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.580451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.579657 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9qnvr\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:56.816892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:56.816801 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:57.164371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:57.164327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:57.286218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:57.285557 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:05:57.293885 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:05:57.293826 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc44cd2_23ff_4641_98d6_0bf07dd75401.slice/crio-5dc8ea92c01431495d9244473313c50931448b69e9a9532774f68d80555ae231 WatchSource:0}: Error finding container 5dc8ea92c01431495d9244473313c50931448b69e9a9532774f68d80555ae231: Status 404 returned error can't find the container with id 5dc8ea92c01431495d9244473313c50931448b69e9a9532774f68d80555ae231 Nov 02 23:05:57.298471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:57.298426 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.298638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:57.298608 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:57.298719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:57.298697 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:05:58.281233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:58.281201 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" event={"ID":"bfc44cd2-23ff-4641-98d6-0bf07dd75401","Type":"ContainerStarted","Data":"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5"} Nov 02 23:05:58.281399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:58.281254 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" event={"ID":"bfc44cd2-23ff-4641-98d6-0bf07dd75401","Type":"ContainerStarted","Data":"5dc8ea92c01431495d9244473313c50931448b69e9a9532774f68d80555ae231"} Nov 02 23:05:58.817624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:58.817595 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:05:58.819162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:58.819140 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:05:58.819265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:58.819207 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:05:59.817479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:59.817446 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:05:59.817710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:05:59.817501 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:00.699993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.699942 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podStartSLOduration=4.6999259559999995 podStartE2EDuration="4.699925956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:58.510140759 +0000 UTC m=+1072.940010619" watchObservedRunningTime="2026-11-02 23:06:00.699925956 +0000 UTC m=+1075.129795816" Nov 02 23:06:00.701882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.701862 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:06:00.705781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.705762 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.763816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.763786 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:06:00.782191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.782148 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.782379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.782307 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.782379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.782341 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.782550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.782477 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrhv\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.811423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.811395 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rqscl\"" Nov 02 23:06:00.817744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.817717 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:06:00.818072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.817759 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:00.883730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.883695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrhv\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.883926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.883748 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.883926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.883806 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.883926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.883829 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.884611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.884569 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.884849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.884824 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.886456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.886431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:00.956165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:00.956093 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrhv\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\") pod \"payload-processing-external-model-87d86bd5c-blk52\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:01.019186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:01.019145 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:01.299280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:01.299197 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:01.427467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:01.427439 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:06:01.427976 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:06:01.427947 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d70890_f12d_4243_bc46_5ef9c31aa782.slice/crio-23f1ad38ad8785be329a8a33e55b2d40c6839a702e216f80dc92d7760ccab650 WatchSource:0}: Error finding container 23f1ad38ad8785be329a8a33e55b2d40c6839a702e216f80dc92d7760ccab650: Status 404 returned error can't find the container with id 23f1ad38ad8785be329a8a33e55b2d40c6839a702e216f80dc92d7760ccab650 Nov 02 23:06:01.818016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:01.817976 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:06:01.818498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:01.818052 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:02.121838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.121756 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:02.271684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.271649 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:02.305587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.305548 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" event={"ID":"40d70890-f12d-4243-bc46-5ef9c31aa782","Type":"ContainerStarted","Data":"a0ced8b0683dd775243f56db8a9956a82b92b2d5313191ac310054c64746a3de"} Nov 02 23:06:02.305808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.305697 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:06:02.305808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.305731 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" event={"ID":"40d70890-f12d-4243-bc46-5ef9c31aa782","Type":"ContainerStarted","Data":"23f1ad38ad8785be329a8a33e55b2d40c6839a702e216f80dc92d7760ccab650"} Nov 02 23:06:02.430209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.428040 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podStartSLOduration=2.4280192720000002 podStartE2EDuration="2.428019272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:02.425593268 +0000 UTC m=+1076.855463129" watchObservedRunningTime="2026-11-02 23:06:02.428019272 +0000 UTC m=+1076.857889134" Nov 02 23:06:02.817969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.817872 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:06:02.817969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:02.817925 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:03.817768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:03.817725 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:06:03.817973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:03.817806 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:04.817602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:04.817557 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:06:04.818011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:04.817624 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 23:06:05.258849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:05.258813 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:06:06.268435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:06.268391 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:06:06.269330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:06.269306 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:06:06.270464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:06.270441 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:06:07.372462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:07.372358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:08.318343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:08.318301 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:06:08.318537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:08.318372 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:06:11.170564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.170530 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:06:11.175681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.175663 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246256 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246294 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246433 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246491 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246585 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246613 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865s4\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246647 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246733 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.246787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.246765 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.271416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.271387 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:06:11.348357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348322 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348398 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348429 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865s4\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348478 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348512 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348539 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348571 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348598 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348721 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.348822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348736 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.349004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.348832 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.349121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.349097 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.349214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.349126 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.349292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.349276 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.351213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.351190 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.351310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.351191 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.376889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.376862 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq"] Nov 02 23:06:11.381901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.381884 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.410165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.410135 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.410301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.410262 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865s4\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\") pod \"maas-default-gateway-openshift-default-694db55c87-q45zr\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.435116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.435044 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:06:11.435351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.435328 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:06:11.437998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.437975 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq"] Nov 02 23:06:11.450105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.450065 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.450326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.450134 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.450326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.450269 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74pg\" (UniqueName: \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.489715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.489681 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:11.551996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.551968 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.552113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.552026 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.552209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.552111 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74pg\" (UniqueName: \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.553618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.553587 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.556211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.556158 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.614613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.614573 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74pg\" (UniqueName: \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\") pod \"maas-api-revoke-sub-b860b205-dvdpq\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.696981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.696911 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:11.921293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.921268 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:06:11.921940 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:06:11.921918 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6965e205_f8fc_4266_b864_c5b3e16f5669.slice/crio-fc0c05d4f65f87838fd78e92e35a714e191db040da471b67ffa85bc98b973525 WatchSource:0}: Error finding container fc0c05d4f65f87838fd78e92e35a714e191db040da471b67ffa85bc98b973525: Status 404 returned error can't find the container with id fc0c05d4f65f87838fd78e92e35a714e191db040da471b67ffa85bc98b973525 Nov 02 23:06:11.924467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.924439 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:11.924528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.924509 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:11.924564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:11.924537 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:12.074434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.074399 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq"] Nov 02 23:06:12.074930 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:06:12.074902 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbdafbe_3408_4de5_a9fe_27bd47cdcacd.slice/crio-e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631 WatchSource:0}: Error finding container e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631: Status 404 returned error can't find the container with id e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631 Nov 02 23:06:12.363871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.363825 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:12.431222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.430488 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" event={"ID":"6965e205-f8fc-4266-b864-c5b3e16f5669","Type":"ContainerStarted","Data":"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce"} Nov 02 23:06:12.431222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.430554 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" event={"ID":"6965e205-f8fc-4266-b864-c5b3e16f5669","Type":"ContainerStarted","Data":"fc0c05d4f65f87838fd78e92e35a714e191db040da471b67ffa85bc98b973525"} Nov 02 23:06:12.436717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.436677 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" event={"ID":"abbdafbe-3408-4de5-a9fe-27bd47cdcacd","Type":"ContainerStarted","Data":"4d7467f0b9c59e43672a8f3d7871627565057def745377c63c02c2bd413bcec2"} Nov 02 23:06:12.436903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.436739 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" event={"ID":"abbdafbe-3408-4de5-a9fe-27bd47cdcacd","Type":"ContainerStarted","Data":"e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631"} Nov 02 23:06:12.492194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.490695 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:12.493052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.493014 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:12.493163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.493082 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:12.679430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.679376 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podStartSLOduration=1.679362228 podStartE2EDuration="1.679362228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:12.562126301 +0000 UTC m=+1086.991996166" watchObservedRunningTime="2026-11-02 23:06:12.679362228 +0000 UTC m=+1087.109232089" Nov 02 23:06:12.679612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:12.679582 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" podStartSLOduration=1.6795773760000001 podStartE2EDuration="1.679577376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:12.67861728 +0000 UTC m=+1087.108487140" watchObservedRunningTime="2026-11-02 23:06:12.679577376 +0000 UTC m=+1087.109447237" Nov 02 23:06:13.315730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:13.315694 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:06:13.461563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:13.461524 4494 generic.go:356] "Generic (PLEG): container finished" podID="abbdafbe-3408-4de5-a9fe-27bd47cdcacd" containerID="4d7467f0b9c59e43672a8f3d7871627565057def745377c63c02c2bd413bcec2" exitCode=0 Nov 02 23:06:13.462201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:13.461591 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" event={"ID":"abbdafbe-3408-4de5-a9fe-27bd47cdcacd","Type":"ContainerDied","Data":"4d7467f0b9c59e43672a8f3d7871627565057def745377c63c02c2bd413bcec2"} Nov 02 23:06:13.490064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:13.490034 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:13.490256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:13.490089 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:14.490308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.490276 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:14.490759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.490341 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:14.664599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.664569 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:14.688619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.688597 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\") pod \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " Nov 02 23:06:14.688729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.688701 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\") pod \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " Nov 02 23:06:14.688764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.688742 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\" (UniqueName: \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\") pod \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\" (UID: \"abbdafbe-3408-4de5-a9fe-27bd47cdcacd\") " Nov 02 23:06:14.689025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.688992 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca" pod "abbdafbe-3408-4de5-a9fe-27bd47cdcacd" (UID: "abbdafbe-3408-4de5-a9fe-27bd47cdcacd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:06:14.690792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.690766 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca" pod "abbdafbe-3408-4de5-a9fe-27bd47cdcacd" (UID: "abbdafbe-3408-4de5-a9fe-27bd47cdcacd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:06:14.690919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.690901 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg" pod "abbdafbe-3408-4de5-a9fe-27bd47cdcacd" (UID: "abbdafbe-3408-4de5-a9fe-27bd47cdcacd"). InnerVolumeSpecName "kube-api-access-d74pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:06:14.790723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.790642 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:06:14.790723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.790678 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:06:14.790723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:14.790695 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74pg\" (UniqueName: \"kubernetes.io/projected/abbdafbe-3408-4de5-a9fe-27bd47cdcacd-kube-api-access-d74pg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:06:15.476730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:15.476702 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" Nov 02 23:06:15.476901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:15.476699 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq" event={"ID":"abbdafbe-3408-4de5-a9fe-27bd47cdcacd","Type":"ContainerDied","Data":"e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631"} Nov 02 23:06:15.476901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:15.476830 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e417b8cb521194482b5a4e815007ca4d77c2083ad8e15d48e69ef1294270e631" Nov 02 23:06:15.490827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:15.490807 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:15.491153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:15.490853 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:16.490149 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:16.490115 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:16.490444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:16.490200 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:17.460353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:17.460284 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:17.501559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:17.501523 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:17.502036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:17.501997 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:17.598757 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:17.598715 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:06:18.490124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:18.490087 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:18.490535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:18.490146 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:19.490296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:19.490265 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 23:06:19.490679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:19.490320 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 23:06:20.569671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:20.569566 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:20.570443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:20.570210 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:20.571307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:20.571286 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:06:22.541490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:22.541443 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:23.313784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:23.313742 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:06:27.058942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:27.058835 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:27.206985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:27.206934 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:27.446640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:27.446609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:30.863386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:30.863342 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:06:32.364749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:32.364713 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:33.314090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:33.314055 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:06:37.270150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:37.270109 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:42.148458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:42.148417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:43.313499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:43.313464 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:06:47.012896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:47.012839 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:47.179269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:47.179227 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:47.559517 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:47.559439 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:06:48.619607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.619573 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:06:48.620145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.620131 4494 cpu_manager.go:414] "containerMap: removing container" podUID="abbdafbe-3408-4de5-a9fe-27bd47cdcacd" containerName="revoke-keys" Nov 02 23:06:48.620238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.620147 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbdafbe-3408-4de5-a9fe-27bd47cdcacd" containerName="revoke-keys" Nov 02 23:06:48.620285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.620274 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbdafbe-3408-4de5-a9fe-27bd47cdcacd" containerName="revoke-keys" Nov 02 23:06:48.625008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.624991 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730481 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730562 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730587 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv879\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730682 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730705 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730741 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730775 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.730940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730849 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.731069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.730946 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.733718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.733691 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:06:48.832167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832136 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832221 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832311 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832368 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832400 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832480 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832505 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv879\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832612 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832617 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832642 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832741 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.832953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.832926 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.833141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.833112 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.833313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.833205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.834663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.834641 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.835062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.835044 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.894550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.894468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.894550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.894492 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv879\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:48.938056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:48.938024 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:49.389015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.388988 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:06:49.389399 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:06:49.389377 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10072372_541a_468f_971f_6c60c604415e.slice/crio-591d840a8c6a8c4f4dabf57c8247c098f3ff9f26cabf5cb8e58cf8fad63b0f51 WatchSource:0}: Error finding container 591d840a8c6a8c4f4dabf57c8247c098f3ff9f26cabf5cb8e58cf8fad63b0f51: Status 404 returned error can't find the container with id 591d840a8c6a8c4f4dabf57c8247c098f3ff9f26cabf5cb8e58cf8fad63b0f51 Nov 02 23:06:49.391870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.391837 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:49.391951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.391905 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:49.391951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.391935 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:06:49.740229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.740188 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" event={"ID":"10072372-541a-468f-971f-6c60c604415e","Type":"ContainerStarted","Data":"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a"} Nov 02 23:06:49.740727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.740253 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" event={"ID":"10072372-541a-468f-971f-6c60c604415e","Type":"ContainerStarted","Data":"591d840a8c6a8c4f4dabf57c8247c098f3ff9f26cabf5cb8e58cf8fad63b0f51"} Nov 02 23:06:49.871751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.871679 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podStartSLOduration=1.8716603109999999 podStartE2EDuration="1.871660311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:49.865529879 +0000 UTC m=+1124.295399741" watchObservedRunningTime="2026-11-02 23:06:49.871660311 +0000 UTC m=+1124.301530172" Nov 02 23:06:49.949080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:49.949031 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:50.955200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:50.948328 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:50.955200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:50.948414 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:51.939510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:51.939476 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:51.939795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:51.939753 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:52.083094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:52.083066 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:52.951198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:52.945574 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:52.951198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:52.945636 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:53.314281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:53.314141 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:06:53.942612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:53.942574 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:53.942907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:53.942875 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:54.724295 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:54.724253 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:54.724295 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:54.724281 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:54.724957 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:54.724347 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:06:55.224331243 +0000 UTC m=+1129.654201082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-cf53d560958d" not found Nov 02 23:06:54.953197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:54.952609 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:54.953197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:54.952673 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:55.228545 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:55.228505 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:55.228545 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:55.228533 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:55.228799 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:55.228589 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:06:56.228572782 +0000 UTC m=+1130.658442621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-cf53d560958d" not found Nov 02 23:06:55.939118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:55.939076 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 23:06:55.939613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:55.939139 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 23:06:56.244904 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:56.244821 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:56.244904 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:56.244846 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:56.244904 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:56.244905 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:06:58.244885431 +0000 UTC m=+1132.674755272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-cf53d560958d" not found Nov 02 23:06:56.943087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:56.943058 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:56.943559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:56.943487 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:56.944245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:56.944226 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:06:57.054671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:57.054639 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:06:57.432768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:57.432734 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:06:58.294186 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:58.294122 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:58.294186 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:58.294163 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-cf53d560958d" not found Nov 02 23:06:58.294792 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:06:58.294274 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:07:02.294252663 +0000 UTC m=+1136.724122522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-cf53d560958d" not found Nov 02 23:06:58.858198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:06:58.844517 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:01.647684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:01.647641 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:02.364287 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:02.364257 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:07:02.364287 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:02.364281 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-cf53d560958d" not found Nov 02 23:07:02.364527 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:02.364335 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:07:10.364320248 +0000 UTC m=+1144.794190087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-cf53d560958d" not found Nov 02 23:07:03.069378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:03.069349 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:03.313306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:03.313272 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:07.237475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.237445 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:07.389270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.389241 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:07:07.390599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.390571 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:07.417439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.417407 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:07:07.422502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.422485 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.501830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.501746 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:07:07.533550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.533513 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.533736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.533580 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnt4z\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.533783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.533757 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.533979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.533957 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.634988 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.635046 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.635064 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt4z\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.635121 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.635731 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.635881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.635827 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.637616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.637598 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.696920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.696892 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt4z\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\") pod \"payload-processing-external-model-745976967d-wqcbc\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:07.735852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:07.735816 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:08.114379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:08.114348 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:07:08.115481 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:07:08.115453 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882d37c1_a117_4fec_afae_f7a9c34436c2.slice/crio-5a3731931769f5b1f3aa91a4d0ce5846b0ce1393175530928458ed9493922b9d WatchSource:0}: Error finding container 5a3731931769f5b1f3aa91a4d0ce5846b0ce1393175530928458ed9493922b9d: Status 404 returned error can't find the container with id 5a3731931769f5b1f3aa91a4d0ce5846b0ce1393175530928458ed9493922b9d Nov 02 23:07:08.884278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:08.884238 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" event={"ID":"882d37c1-a117-4fec-afae-f7a9c34436c2","Type":"ContainerStarted","Data":"e50f6236cb4c9f01c7696c3dc6d8f19f211494a752eae699d96ef6dadb282aff"} Nov 02 23:07:08.884718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:08.884303 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" event={"ID":"882d37c1-a117-4fec-afae-f7a9c34436c2","Type":"ContainerStarted","Data":"5a3731931769f5b1f3aa91a4d0ce5846b0ce1393175530928458ed9493922b9d"} Nov 02 23:07:08.884718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:08.884437 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:07:09.004621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:09.004569 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podStartSLOduration=2.004553477 podStartE2EDuration="2.004553477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:09.002024864 +0000 UTC m=+1143.431894726" watchObservedRunningTime="2026-11-02 23:07:09.004553477 +0000 UTC m=+1143.434423338" Nov 02 23:07:10.370711 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:10.370678 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8a8e5c7c40ee: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:10.370711 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:10.370712 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:10.371142 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:10.370775 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:07:26.370761353 +0000 UTC m=+1160.800631206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:12.389718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:12.389661 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" containerID="cri-o://a0ced8b0683dd775243f56db8a9956a82b92b2d5313191ac310054c64746a3de" gracePeriod=25 Nov 02 23:07:12.463213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:12.463154 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:13.018970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.018930 4494 generic.go:356] "Generic (PLEG): container finished" podID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerID="a0ced8b0683dd775243f56db8a9956a82b92b2d5313191ac310054c64746a3de" exitCode=0 Nov 02 23:07:13.019330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.019303 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" event={"ID":"40d70890-f12d-4243-bc46-5ef9c31aa782","Type":"ContainerDied","Data":"a0ced8b0683dd775243f56db8a9956a82b92b2d5313191ac310054c64746a3de"} Nov 02 23:07:13.218254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.217604 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:07:13.300770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.300718 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\") pod \"40d70890-f12d-4243-bc46-5ef9c31aa782\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " Nov 02 23:07:13.301103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.301082 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\") pod \"40d70890-f12d-4243-bc46-5ef9c31aa782\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " Nov 02 23:07:13.301450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.301428 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\") pod \"40d70890-f12d-4243-bc46-5ef9c31aa782\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " Nov 02 23:07:13.301601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.301584 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\") pod \"40d70890-f12d-4243-bc46-5ef9c31aa782\" (UID: \"40d70890-f12d-4243-bc46-5ef9c31aa782\") " Nov 02 23:07:13.306501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.306475 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume" pod "40d70890-f12d-4243-bc46-5ef9c31aa782" (UID: "40d70890-f12d-4243-bc46-5ef9c31aa782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:13.307548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.307526 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume" pod "40d70890-f12d-4243-bc46-5ef9c31aa782" (UID: "40d70890-f12d-4243-bc46-5ef9c31aa782"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:13.320230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.320199 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv" pod "40d70890-f12d-4243-bc46-5ef9c31aa782" (UID: "40d70890-f12d-4243-bc46-5ef9c31aa782"). InnerVolumeSpecName "kube-api-access-bsrhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:13.326696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.326667 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials" pod "40d70890-f12d-4243-bc46-5ef9c31aa782" (UID: "40d70890-f12d-4243-bc46-5ef9c31aa782"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:13.403852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.403816 4494 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-routing-overlay-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:07:13.439923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.439895 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70890-f12d-4243-bc46-5ef9c31aa782-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:07:13.494257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.484036 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrhv\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-kube-api-access-bsrhv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:07:13.494257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:13.484117 4494 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40d70890-f12d-4243-bc46-5ef9c31aa782-provider-credentials\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:07:14.068255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:14.067409 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" event={"ID":"40d70890-f12d-4243-bc46-5ef9c31aa782","Type":"ContainerDied","Data":"23f1ad38ad8785be329a8a33e55b2d40c6839a702e216f80dc92d7760ccab650"} Nov 02 23:07:14.068255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:14.067484 4494 scope.go:122] "RemoveContainer" containerID="a0ced8b0683dd775243f56db8a9956a82b92b2d5313191ac310054c64746a3de" Nov 02 23:07:14.068255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:14.067647 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87d86bd5c-blk52" Nov 02 23:07:14.295205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:14.291796 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:07:14.344251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:14.343297 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-87d86bd5c-blk52"] Nov 02 23:07:15.693087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:15.693052 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" path="/var/lib/kubelet/pods/40d70890-f12d-4243-bc46-5ef9c31aa782/volumes" Nov 02 23:07:17.573475 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:17.573435 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:07:17.646253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:17.646210 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:19.893399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:19.893355 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:22.530288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.530251 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:07:22.530788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.530774 4494 cpu_manager.go:414] "containerMap: removing container" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" Nov 02 23:07:22.530788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.530788 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" Nov 02 23:07:22.530917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.530908 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d70890-f12d-4243-bc46-5ef9c31aa782" containerName="payload-processing" Nov 02 23:07:22.536051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.536029 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.549379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.549345 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:22.592645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.592618 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:07:22.639540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.639512 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"kube-root-ca.crt\"" Nov 02 23:07:22.639540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.639540 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"openshift-service-ca.crt\"" Nov 02 23:07:22.639752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.639537 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"e2e-conflict-b-4cba9672-kserve-self-signed-certs\"" Nov 02 23:07:22.639752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.639547 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-08c2c6\"/\"default-dockercfg-zg2jb\"" Nov 02 23:07:22.708033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.707903 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:22.738265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.720893 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.738642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.738604 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.738871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.738851 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.739055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.739034 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.739341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.739319 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.739545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.739524 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.840965 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841070 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841107 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841136 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841232 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841279 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841412 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841647 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.841880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.841689 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.843755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.843709 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.844212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.844190 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:22.903339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:22.903299 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\") pod \"e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:23.152299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:23.152160 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:23.529776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:23.529747 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:07:23.530800 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:07:23.530770 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafdf9b4e_a2da_442e_9a85_c938083c2e9b.slice/crio-a36364ecf53eab1239ed8a97d2ea4f781df5564ff717b6985e62279a3fbdbdcf WatchSource:0}: Error finding container a36364ecf53eab1239ed8a97d2ea4f781df5564ff717b6985e62279a3fbdbdcf: Status 404 returned error can't find the container with id a36364ecf53eab1239ed8a97d2ea4f781df5564ff717b6985e62279a3fbdbdcf Nov 02 23:07:24.130859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.130830 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:07:24.135202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.135153 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.139494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.139464 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" event={"ID":"afdf9b4e-a2da-442e-9a85-c938083c2e9b","Type":"ContainerStarted","Data":"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14"} Nov 02 23:07:24.139635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.139505 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" event={"ID":"afdf9b4e-a2da-442e-9a85-c938083c2e9b","Type":"ContainerStarted","Data":"a36364ecf53eab1239ed8a97d2ea4f781df5564ff717b6985e62279a3fbdbdcf"} Nov 02 23:07:24.153064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.153022 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:24.173704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.173680 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.173824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.173729 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.173891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.173873 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776n2\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.173942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.173905 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.174031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.174017 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.174070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.174050 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.174120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.174073 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.174161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.174129 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.174262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.174250 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.232078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.232040 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:07:24.275026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.274999 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275059 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275094 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275117 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275208 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776n2\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275227 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275288 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275364 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275484 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275741 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.275844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.275821 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.276113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.276087 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.276260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.276235 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.277565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.277542 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.277907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.277890 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.337529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.337500 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.337685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.337663 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776n2\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\") pod \"e2e-worker-w1-b42983-openshift-default-c98888847-sc27j\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.373753 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.373702 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" podStartSLOduration=2.373686082 podStartE2EDuration="2.373686082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:24.373242053 +0000 UTC m=+1158.803111921" watchObservedRunningTime="2026-11-02 23:07:24.373686082 +0000 UTC m=+1158.803555955" Nov 02 23:07:24.449923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.449888 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:24.952594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.952562 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:07:24.953051 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:07:24.953028 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723a6748_877b_43d4_ac76_0e14380a2cc3.slice/crio-09e1f8910ba0176b7d4f466898e873e12c79317a9326d140b8e54c98bebcf297 WatchSource:0}: Error finding container 09e1f8910ba0176b7d4f466898e873e12c79317a9326d140b8e54c98bebcf297: Status 404 returned error can't find the container with id 09e1f8910ba0176b7d4f466898e873e12c79317a9326d140b8e54c98bebcf297 Nov 02 23:07:24.956113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.956071 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:07:24.956222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.956155 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:07:24.956222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:24.956208 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:07:25.147204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:25.147158 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" event={"ID":"723a6748-877b-43d4-ac76-0e14380a2cc3","Type":"ContainerStarted","Data":"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937"} Nov 02 23:07:25.147380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:25.147226 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" event={"ID":"723a6748-877b-43d4-ac76-0e14380a2cc3","Type":"ContainerStarted","Data":"09e1f8910ba0176b7d4f466898e873e12c79317a9326d140b8e54c98bebcf297"} Nov 02 23:07:25.276294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:25.276161 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podStartSLOduration=1.276142802 podStartE2EDuration="1.276142802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:25.274043953 +0000 UTC m=+1159.703913815" watchObservedRunningTime="2026-11-02 23:07:25.276142802 +0000 UTC m=+1159.706012689" Nov 02 23:07:25.451324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:25.451281 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:26.403726 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:26.403698 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8a8e5c7c40ee: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:26.403726 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:26.403723 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:26.404222 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:26.403777 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:07:58.40376248 +0000 UTC m=+1192.833632335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:26.457295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:26.457251 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:26.457515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:26.457331 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:27.451200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:27.450623 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:27.451200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:27.450685 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:27.564831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:27.564787 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:28.455201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:28.454962 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:28.455201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:28.455022 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:29.455252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:29.454624 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:29.455252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:29.454687 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:29.455843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:29.455784 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:29.865890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:29.865799 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:07:29.891470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:29.891430 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:30.455196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:30.450656 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:30.455196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:30.450724 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:31.450793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:31.450753 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 23:07:31.451013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:31.450816 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 23:07:32.454291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:32.454262 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:32.454710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:32.454690 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:32.455458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:32.455438 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:07:32.575671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:32.575632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:33.152982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:33.152951 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:33.167655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:33.167624 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:33.278824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:33.278797 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:07:37.551674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:37.551644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:38.238291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:38.238260 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:39.891883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:39.891850 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:42.457068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:42.457024 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:47.365484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:47.365442 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:47.561611 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:47.561569 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:07:49.891833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:49.891797 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:07:49.969061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:49.969030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:52.352730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:52.352690 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:56.587081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:56.587050 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:07:57.263912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:57.263869 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:07:58.434114 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:58.434073 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-8a8e5c7c40ee: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:58.434114 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:58.434111 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz: secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:58.434622 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:07:58.434209 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials podName:c86713c3-09ba-40e7-b039-2617ae3f7940 nodeName:}" failed. No retries permitted until 2026-11-02 23:09:02.434163412 +0000 UTC m=+1256.864033251 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials") pod "payload-processing-external-model-78cb97dd58-7h4cz" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940") : secret "agc-cred-8a8e5c7c40ee" not found Nov 02 23:07:59.892360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:07:59.892329 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:08:01.991078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:01.991030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:02.159580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.159538 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:02.709446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.709409 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:08:02.713866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.713844 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.816954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.816923 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:08:02.888671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.888850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888698 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.888850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888835 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.888934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888862 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.888934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888884 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.889019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.888998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.889055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.889036 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvjh\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.889133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.889119 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.889199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.889143 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990541 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990582 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvjh\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990640 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990659 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990700 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990748 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990803 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.990844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990819 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.990933 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.991051 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.991082 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.991147 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.991258 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.991520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.991492 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.993269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.993243 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:02.993583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:02.993561 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:03.052055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.052022 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:03.052244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.052092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvjh\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:03.327502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.327420 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:03.756231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.756200 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:08:03.793008 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:08:03.792979 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e06b5f_1966_4696_8a0c_b355d63692b1.slice/crio-20b336f8f7465d5815223fd991c651057dd437a34dbda1d2e7f50ec6a8098be0 WatchSource:0}: Error finding container 20b336f8f7465d5815223fd991c651057dd437a34dbda1d2e7f50ec6a8098be0: Status 404 returned error can't find the container with id 20b336f8f7465d5815223fd991c651057dd437a34dbda1d2e7f50ec6a8098be0 Nov 02 23:08:03.795350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.795322 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:03.795435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.795388 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:03.795435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:03.795417 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:08:04.518347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:04.518305 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" event={"ID":"d1e06b5f-1966-4696-8a0c-b355d63692b1","Type":"ContainerStarted","Data":"0205842b73be582aa8f0c8aecfd9a603dfd57650e74ee21216a3fa7f534184e4"} Nov 02 23:08:04.518770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:04.518369 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" event={"ID":"d1e06b5f-1966-4696-8a0c-b355d63692b1","Type":"ContainerStarted","Data":"20b336f8f7465d5815223fd991c651057dd437a34dbda1d2e7f50ec6a8098be0"} Nov 02 23:08:04.728519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:04.728467 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podStartSLOduration=2.728451594 podStartE2EDuration="2.728451594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:04.689987608 +0000 UTC m=+1199.119857470" watchObservedRunningTime="2026-11-02 23:08:04.728451594 +0000 UTC m=+1199.158321455" Nov 02 23:08:05.328039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.327997 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:05.329379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.329353 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:05.329505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.329406 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:05.910897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.910865 4494 scope.go:122] "RemoveContainer" containerID="f9f9c6949616d0c3f5af722ce90e5bd999a2fee7aacd070ca6988e1e0be8bc5c" Nov 02 23:08:05.920444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.920421 4494 scope.go:122] "RemoveContainer" containerID="a8678d7be255420b5cdb7b3624c048b89b3311b65a386987ad9f7aa86274a486" Nov 02 23:08:05.929085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.929065 4494 scope.go:122] "RemoveContainer" containerID="6cad3909c3928b684b740671305a03e0625845a11bb3b0e7a8048fd83f7f80e2" Nov 02 23:08:05.963367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.963339 4494 scope.go:122] "RemoveContainer" containerID="b5aadd8ae01da3ae0a6ab4eed93177d16e97789c6053a405e01b46856f4b04f1" Nov 02 23:08:05.972204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:05.972166 4494 scope.go:122] "RemoveContainer" containerID="b2c4922450639e6976ba83b376b2ba93bdd64f170c408374321171b633d7511a" Nov 02 23:08:06.328606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:06.328524 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:06.328606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:06.328589 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:07.185084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:07.185047 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:07.328660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:07.328629 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:07.328826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:07.328687 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:08.328750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:08.328716 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:08.329146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:08.328770 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:09.328814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:09.328780 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:09.329256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:09.328842 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:09.891556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:09.891521 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:08:10.328205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:10.328141 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:08:10.328394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:10.328237 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 23:08:11.332683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:11.332648 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:11.333085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:11.332907 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:11.333849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:11.333827 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:08:11.760411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:11.760307 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:12.175872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:12.175846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:14.018965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:14.018926 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:14.167694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:14.167651 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:14.314439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:14.314341 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:17.159087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:17.159058 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:17.561796 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:08:17.561676 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:08:18.463800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.463762 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:08:18.471462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.471427 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:08:18.508531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.508494 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:08:18.514624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.514602 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.572373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.572321 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-832363c1011a\"" Nov 02 23:08:18.572649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.572403 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-91389874e6b5\"" Nov 02 23:08:18.578213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.578186 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:08:18.636986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.636954 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.637213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.637080 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tsm\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.637213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.637144 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.637415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.637395 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.738407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.738317 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tsm\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.738407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.738374 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.738627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.738496 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.738627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.738541 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.739190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.739150 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.739190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.739163 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.741295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.741275 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.800446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.800411 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tsm\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\") pod \"payload-processing-external-model-5969b46dfd-d2ppj\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:18.827897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:18.827868 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:19.228898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:19.228868 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:08:19.230132 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:08:19.230099 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9325081_1096_49b3_a093_24a6916bd9af.slice/crio-d8d65610842a2afb1c3b1b3b7c4d0ce93645fb906ab97832873408c45278a2dd WatchSource:0}: Error finding container d8d65610842a2afb1c3b1b3b7c4d0ce93645fb906ab97832873408c45278a2dd: Status 404 returned error can't find the container with id d8d65610842a2afb1c3b1b3b7c4d0ce93645fb906ab97832873408c45278a2dd Nov 02 23:08:19.666994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:19.666965 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" event={"ID":"e9325081-1096-49b3-a093-24a6916bd9af","Type":"ContainerStarted","Data":"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb"} Nov 02 23:08:19.667409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:19.667013 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" event={"ID":"e9325081-1096-49b3-a093-24a6916bd9af","Type":"ContainerStarted","Data":"d8d65610842a2afb1c3b1b3b7c4d0ce93645fb906ab97832873408c45278a2dd"} Nov 02 23:08:19.667409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:19.667108 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:19.785493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:19.785438 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" podStartSLOduration=1.785422687 podStartE2EDuration="1.785422687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:19.784989947 +0000 UTC m=+1214.214859808" watchObservedRunningTime="2026-11-02 23:08:19.785422687 +0000 UTC m=+1214.215292554" Nov 02 23:08:22.138833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:22.138803 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:23.465228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.465189 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" containerID="cri-o://e50f6236cb4c9f01c7696c3dc6d8f19f211494a752eae699d96ef6dadb282aff" gracePeriod=25 Nov 02 23:08:23.696345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.696314 4494 generic.go:356] "Generic (PLEG): container finished" podID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerID="e50f6236cb4c9f01c7696c3dc6d8f19f211494a752eae699d96ef6dadb282aff" exitCode=0 Nov 02 23:08:23.696345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.696349 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" event={"ID":"882d37c1-a117-4fec-afae-f7a9c34436c2","Type":"ContainerDied","Data":"e50f6236cb4c9f01c7696c3dc6d8f19f211494a752eae699d96ef6dadb282aff"} Nov 02 23:08:23.726630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.726568 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:08:23.811888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.811848 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\") pod \"882d37c1-a117-4fec-afae-f7a9c34436c2\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " Nov 02 23:08:23.812086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.811954 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\") pod \"882d37c1-a117-4fec-afae-f7a9c34436c2\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " Nov 02 23:08:23.812086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812031 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\") pod \"882d37c1-a117-4fec-afae-f7a9c34436c2\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " Nov 02 23:08:23.812253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812095 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\") pod \"882d37c1-a117-4fec-afae-f7a9c34436c2\" (UID: \"882d37c1-a117-4fec-afae-f7a9c34436c2\") " Nov 02 23:08:23.812422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812392 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume" pod "882d37c1-a117-4fec-afae-f7a9c34436c2" (UID: "882d37c1-a117-4fec-afae-f7a9c34436c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:08:23.812570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812428 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume" pod "882d37c1-a117-4fec-afae-f7a9c34436c2" (UID: "882d37c1-a117-4fec-afae-f7a9c34436c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:08:23.812777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812711 4494 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-routing-overlay-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:23.812907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.812865 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d37c1-a117-4fec-afae-f7a9c34436c2-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:23.814238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.814208 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z" pod "882d37c1-a117-4fec-afae-f7a9c34436c2" (UID: "882d37c1-a117-4fec-afae-f7a9c34436c2"). InnerVolumeSpecName "kube-api-access-jnt4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:08:23.814354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.814258 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials" pod "882d37c1-a117-4fec-afae-f7a9c34436c2" (UID: "882d37c1-a117-4fec-afae-f7a9c34436c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:08:23.914140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.914107 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnt4z\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-kube-api-access-jnt4z\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:23.914140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:23.914137 4494 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882d37c1-a117-4fec-afae-f7a9c34436c2-provider-credentials\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:24.506551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.506517 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:08:24.703980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.703953 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" Nov 02 23:08:24.703980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.703957 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-745976967d-wqcbc" event={"ID":"882d37c1-a117-4fec-afae-f7a9c34436c2","Type":"ContainerDied","Data":"5a3731931769f5b1f3aa91a4d0ce5846b0ce1393175530928458ed9493922b9d"} Nov 02 23:08:24.704235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.704015 4494 scope.go:122] "RemoveContainer" containerID="e50f6236cb4c9f01c7696c3dc6d8f19f211494a752eae699d96ef6dadb282aff" Nov 02 23:08:24.879957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.879873 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:08:24.883698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:24.883677 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-745976967d-wqcbc"] Nov 02 23:08:25.774198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:25.759841 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" path="/var/lib/kubelet/pods/882d37c1-a117-4fec-afae-f7a9c34436c2/volumes" Nov 02 23:08:29.073197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:29.072301 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:30.676509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:30.676485 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:08:30.850548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:30.850511 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:08:32.143400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:32.143360 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:34.069899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:34.069837 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:35.851143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:35.851082 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" podUID="c86713c3-09ba-40e7-b039-2617ae3f7940" containerName="payload-processing" containerID="cri-o://9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983" gracePeriod=25 Nov 02 23:08:36.110027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.109925 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:08:36.283193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283150 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\") pod \"c86713c3-09ba-40e7-b039-2617ae3f7940\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " Nov 02 23:08:36.283373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283269 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\") pod \"c86713c3-09ba-40e7-b039-2617ae3f7940\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " Nov 02 23:08:36.283373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283318 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\") pod \"c86713c3-09ba-40e7-b039-2617ae3f7940\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " Nov 02 23:08:36.283480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283414 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\") pod \"c86713c3-09ba-40e7-b039-2617ae3f7940\" (UID: \"c86713c3-09ba-40e7-b039-2617ae3f7940\") " Nov 02 23:08:36.283692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283668 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume" pod "c86713c3-09ba-40e7-b039-2617ae3f7940" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:08:36.283799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283692 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume" pod "c86713c3-09ba-40e7-b039-2617ae3f7940" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:08:36.283863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283849 4494 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-routing-overlay-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:36.283914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.283866 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86713c3-09ba-40e7-b039-2617ae3f7940-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:36.285713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.285679 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb" pod "c86713c3-09ba-40e7-b039-2617ae3f7940" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940"). InnerVolumeSpecName "kube-api-access-ghbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:08:36.285811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.285773 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials" pod "c86713c3-09ba-40e7-b039-2617ae3f7940" (UID: "c86713c3-09ba-40e7-b039-2617ae3f7940"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:08:36.385346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.385264 4494 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-provider-credentials\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:36.385346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.385293 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbgb\" (UniqueName: \"kubernetes.io/projected/c86713c3-09ba-40e7-b039-2617ae3f7940-kube-api-access-ghbgb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:08:36.893339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.893300 4494 generic.go:356] "Generic (PLEG): container finished" podID="c86713c3-09ba-40e7-b039-2617ae3f7940" containerID="9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983" exitCode=0 Nov 02 23:08:36.893832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.893344 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" Nov 02 23:08:36.893832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.893385 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" event={"ID":"c86713c3-09ba-40e7-b039-2617ae3f7940","Type":"ContainerDied","Data":"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983"} Nov 02 23:08:36.893832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.893441 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz" event={"ID":"c86713c3-09ba-40e7-b039-2617ae3f7940","Type":"ContainerDied","Data":"308bdb7187316b6ff2b5a8bb6b12e565ca1340f1c87f04ffa8e5463a13691386"} Nov 02 23:08:36.893832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.893467 4494 scope.go:122] "RemoveContainer" containerID="9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983" Nov 02 23:08:36.917075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.917047 4494 scope.go:122] "RemoveContainer" containerID="9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983" Nov 02 23:08:36.917636 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:08:36.917615 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983\": container with ID starting with 9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983 not found: ID does not exist" containerID="9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983" Nov 02 23:08:36.917704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:36.917645 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983"} err="failed to get container status \"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983\": rpc error: code = NotFound desc = could not find container \"9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983\": container with ID starting with 9d7cd6edb2c5a0238e09b86e0a9919ab03164386b36834dbd1ceba0593991983 not found: ID does not exist" Nov 02 23:08:37.061607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:37.061554 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:08:37.111781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:37.111749 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78cb97dd58-7h4cz"] Nov 02 23:08:37.694928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:37.694893 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86713c3-09ba-40e7-b039-2617ae3f7940" path="/var/lib/kubelet/pods/c86713c3-09ba-40e7-b039-2617ae3f7940/volumes" Nov 02 23:08:39.059022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:39.058986 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:44.777234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:44.777200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:47.559464 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:08:47.559423 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:08:47.963676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:47.963647 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:08:47.966592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:47.966566 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:49.784844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:49.784794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:54.683682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:54.683647 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:08:58.906426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.906390 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5"] Nov 02 23:08:58.906961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.906949 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c86713c3-09ba-40e7-b039-2617ae3f7940" containerName="payload-processing" Nov 02 23:08:58.907013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.906962 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86713c3-09ba-40e7-b039-2617ae3f7940" containerName="payload-processing" Nov 02 23:08:58.907013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.906981 4494 cpu_manager.go:414] "containerMap: removing container" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" Nov 02 23:08:58.907013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.906986 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" Nov 02 23:08:58.907120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.907063 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86713c3-09ba-40e7-b039-2617ae3f7940" containerName="payload-processing" Nov 02 23:08:58.907120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.907074 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="882d37c1-a117-4fec-afae-f7a9c34436c2" containerName="payload-processing" Nov 02 23:08:58.910616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.910598 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:58.954485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.954449 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5"] Nov 02 23:08:58.964117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.964086 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:08:58.964439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:58.964405 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:08:59.003682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.003654 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.003822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.003720 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hn9q\" (UniqueName: \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.003868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.003839 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.020337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.020310 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:08:59.105398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.105373 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.105616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.105445 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hn9q\" (UniqueName: \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.105616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.105476 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.106206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.106150 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.108537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.108511 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.167567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.167537 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hn9q\" (UniqueName: \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\") pod \"maas-api-revoke-sub-44387080-8g4p5\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.225347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.225314 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:08:59.618941 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:08:59.618914 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e07e292_be6c_4752_8e1e_d7204949ddb6.slice/crio-42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0 WatchSource:0}: Error finding container 42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0: Status 404 returned error can't find the container with id 42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0 Nov 02 23:08:59.621024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.620996 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5"] Nov 02 23:08:59.659043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:08:59.659011 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:00.257272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:00.257236 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" event={"ID":"2e07e292-be6c-4752-8e1e-d7204949ddb6","Type":"ContainerStarted","Data":"4ea10cc1126b31804ce9b7c416357b8f28ab1a3ce5e20919de014bffe3a4908b"} Nov 02 23:09:00.257770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:00.257299 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" event={"ID":"2e07e292-be6c-4752-8e1e-d7204949ddb6","Type":"ContainerStarted","Data":"42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0"} Nov 02 23:09:00.378835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:00.378713 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" podStartSLOduration=2.3786962689999998 podStartE2EDuration="2.378696269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:00.374123512 +0000 UTC m=+1254.803993374" watchObservedRunningTime="2026-11-02 23:09:00.378696269 +0000 UTC m=+1254.808566131" Nov 02 23:09:01.264630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:01.264599 4494 generic.go:356] "Generic (PLEG): container finished" podID="2e07e292-be6c-4752-8e1e-d7204949ddb6" containerID="4ea10cc1126b31804ce9b7c416357b8f28ab1a3ce5e20919de014bffe3a4908b" exitCode=0 Nov 02 23:09:01.265021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:01.264693 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" event={"ID":"2e07e292-be6c-4752-8e1e-d7204949ddb6","Type":"ContainerDied","Data":"4ea10cc1126b31804ce9b7c416357b8f28ab1a3ce5e20919de014bffe3a4908b"} Nov 02 23:09:02.342246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.342203 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:02.419288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.419258 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:09:02.556937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.556857 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\") pod \"2e07e292-be6c-4752-8e1e-d7204949ddb6\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " Nov 02 23:09:02.556937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.556926 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\") pod \"2e07e292-be6c-4752-8e1e-d7204949ddb6\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " Nov 02 23:09:02.557130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.556964 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\" (UniqueName: \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\") pod \"2e07e292-be6c-4752-8e1e-d7204949ddb6\" (UID: \"2e07e292-be6c-4752-8e1e-d7204949ddb6\") " Nov 02 23:09:02.557342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.557316 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca" pod "2e07e292-be6c-4752-8e1e-d7204949ddb6" (UID: "2e07e292-be6c-4752-8e1e-d7204949ddb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:02.557574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.557536 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:02.559219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.559143 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca" pod "2e07e292-be6c-4752-8e1e-d7204949ddb6" (UID: "2e07e292-be6c-4752-8e1e-d7204949ddb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:02.559306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.559274 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q" pod "2e07e292-be6c-4752-8e1e-d7204949ddb6" (UID: "2e07e292-be6c-4752-8e1e-d7204949ddb6"). InnerVolumeSpecName "kube-api-access-6hn9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:02.659116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.659078 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e07e292-be6c-4752-8e1e-d7204949ddb6-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:02.659116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.659112 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hn9q\" (UniqueName: \"kubernetes.io/projected/2e07e292-be6c-4752-8e1e-d7204949ddb6-kube-api-access-6hn9q\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:02.984729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:02.984683 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" podUID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" containerName="main" containerID="cri-o://57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14" gracePeriod=45 Nov 02 23:09:03.235644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.235571 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:09:03.279407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.279377 4494 generic.go:356] "Generic (PLEG): container finished" podID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" containerID="57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14" exitCode=0 Nov 02 23:09:03.279608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.279423 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" Nov 02 23:09:03.279608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.279463 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" event={"ID":"afdf9b4e-a2da-442e-9a85-c938083c2e9b","Type":"ContainerDied","Data":"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14"} Nov 02 23:09:03.279608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.279509 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2" event={"ID":"afdf9b4e-a2da-442e-9a85-c938083c2e9b","Type":"ContainerDied","Data":"a36364ecf53eab1239ed8a97d2ea4f781df5564ff717b6985e62279a3fbdbdcf"} Nov 02 23:09:03.279608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.279530 4494 scope.go:122] "RemoveContainer" containerID="57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14" Nov 02 23:09:03.281040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.281014 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" Nov 02 23:09:03.281132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.281055 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5" event={"ID":"2e07e292-be6c-4752-8e1e-d7204949ddb6","Type":"ContainerDied","Data":"42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0"} Nov 02 23:09:03.281132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.281117 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ec01cf738892886d4793cbf741a57d198f70d89c9c797afa015c18113d5ac0" Nov 02 23:09:03.290306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.290286 4494 scope.go:122] "RemoveContainer" containerID="57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14" Nov 02 23:09:03.290594 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:09:03.290576 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14\": container with ID starting with 57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14 not found: ID does not exist" containerID="57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14" Nov 02 23:09:03.290650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.290604 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14"} err="failed to get container status \"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14\": rpc error: code = NotFound desc = could not find container \"57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14\": container with ID starting with 57de39a5f2b08045af92669baa3dfe8843c0cff84ead15d5e9c78156ce8beb14 not found: ID does not exist" Nov 02 23:09:03.368347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368318 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368377 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368402 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368509 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368643 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368671 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\") pod \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\" (UID: \"afdf9b4e-a2da-442e-9a85-c938083c2e9b\") " Nov 02 23:09:03.368855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368763 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:03.369147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.368943 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:03.369415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.369392 4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-home\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.369491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.369422 4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tmp-dir\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.369491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.369443 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:03.370959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.370933 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:03.371060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.370940 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "kube-api-access-cr627". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:03.371060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.370977 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm" pod "afdf9b4e-a2da-442e-9a85-c938083c2e9b" (UID: "afdf9b4e-a2da-442e-9a85-c938083c2e9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:09:03.470536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.470505 4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-model-cache\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.470536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.470531 4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afdf9b4e-a2da-442e-9a85-c938083c2e9b-dshm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.470536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.470541 4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afdf9b4e-a2da-442e-9a85-c938083c2e9b-tls-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.470768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.470550 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr627\" (UniqueName: \"kubernetes.io/projected/afdf9b4e-a2da-442e-9a85-c938083c2e9b-kube-api-access-cr627\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:03.701731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.701694 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:09:03.707509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:03.707484 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-08c2c6/e2e-conflict-b-4cba9672-kserve-6d9557cff5-vfbs2"] Nov 02 23:09:04.562116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.562039 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:04.679014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.678976 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:09:04.680089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680061 4494 cpu_manager.go:414] "containerMap: removing container" podUID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" containerName="main" Nov 02 23:09:04.680344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680323 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" containerName="main" Nov 02 23:09:04.680530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680512 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2e07e292-be6c-4752-8e1e-d7204949ddb6" containerName="revoke-keys" Nov 02 23:09:04.680530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680529 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e07e292-be6c-4752-8e1e-d7204949ddb6" containerName="revoke-keys" Nov 02 23:09:04.680722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680709 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e07e292-be6c-4752-8e1e-d7204949ddb6" containerName="revoke-keys" Nov 02 23:09:04.680835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.680825 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" containerName="main" Nov 02 23:09:04.689026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.688999 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785453 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785508 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785591 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785706 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785747 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.785808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785776 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.786005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.785970 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.786152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.786061 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ntz\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.786283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.786164 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.887148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.887071 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.887148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.887139 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.887392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.887211 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.887774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.887610 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.887934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.887789 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888377 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888455 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888481 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888487 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888543 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.888685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.888659 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ntz\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.889284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.889044 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.889284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.889092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.889284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.889120 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.891592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.891567 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.892002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.891983 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.893104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.893074 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:09:04.950779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.950742 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:04.950917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:04.950828 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ntz\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:05.003864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.003835 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:05.435516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.435489 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:09:05.436436 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:05.436411 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931eb30c_36be_49cf_8b2f_25f5c5bf1497.slice/crio-43e176dc32c4aa7e5fb828ec3d0f10595be1faea3f714da7083314c5741600cd WatchSource:0}: Error finding container 43e176dc32c4aa7e5fb828ec3d0f10595be1faea3f714da7083314c5741600cd: Status 404 returned error can't find the container with id 43e176dc32c4aa7e5fb828ec3d0f10595be1faea3f714da7083314c5741600cd Nov 02 23:09:05.438933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.438905 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:05.439002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.438972 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:05.439037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.438999 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:05.695875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:05.695792 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdf9b4e-a2da-442e-9a85-c938083c2e9b" path="/var/lib/kubelet/pods/afdf9b4e-a2da-442e-9a85-c938083c2e9b/volumes" Nov 02 23:09:06.302134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:06.302038 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" event={"ID":"931eb30c-36be-49cf-8b2f-25f5c5bf1497","Type":"ContainerStarted","Data":"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d"} Nov 02 23:09:06.302312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:06.302136 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" event={"ID":"931eb30c-36be-49cf-8b2f-25f5c5bf1497","Type":"ContainerStarted","Data":"43e176dc32c4aa7e5fb828ec3d0f10595be1faea3f714da7083314c5741600cd"} Nov 02 23:09:06.476566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:06.476509 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podStartSLOduration=2.476493922 podStartE2EDuration="2.476493922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:06.473715326 +0000 UTC m=+1260.903585190" watchObservedRunningTime="2026-11-02 23:09:06.476493922 +0000 UTC m=+1260.906363782" Nov 02 23:09:07.004261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.004220 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:07.005995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.005964 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:07.006120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.006013 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:07.121022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.120988 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 02 23:09:07.126400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.126378 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.180388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.179783 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-96031a\"/\"distinct-2-w3-96031a-kserve-self-signed-certs\"" Nov 02 23:09:07.184741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.184495 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 02 23:09:07.210523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.210488 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:07.217870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.217844 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.218032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.217956 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.218032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.218008 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.218130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.218094 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdln\" (UniqueName: \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.218185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.218132 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.218185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.218147 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.319208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.319087 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.319595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.319551 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.320971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.319765 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.321202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.321138 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdln\" (UniqueName: \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.321294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.321242 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.321294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.321271 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.325212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.321780 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.325212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.322101 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.325212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.324484 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.325674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.325655 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.329147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.329122 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.397287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.397254 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdln\" (UniqueName: \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\") pod \"distinct-2-w3-96031a-kserve-8b66966b9-ndwr7\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.465568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.465533 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:07.993681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:07.993652 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 02 23:09:07.993965 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:07.993930 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243999e3_7f65_4446_930d_b4eb8d180be0.slice/crio-43d7238b3a4acb1556446464596c3ea2b48100707d24afec6334ac61cc46a1a1 WatchSource:0}: Error finding container 43d7238b3a4acb1556446464596c3ea2b48100707d24afec6334ac61cc46a1a1: Status 404 returned error can't find the container with id 43d7238b3a4acb1556446464596c3ea2b48100707d24afec6334ac61cc46a1a1 Nov 02 23:09:08.004381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.004355 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:08.004717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.004413 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:08.345254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.331941 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" event={"ID":"243999e3-7f65-4446-930d-b4eb8d180be0","Type":"ContainerStarted","Data":"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e"} Nov 02 23:09:08.345254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.331989 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" event={"ID":"243999e3-7f65-4446-930d-b4eb8d180be0","Type":"ContainerStarted","Data":"43d7238b3a4acb1556446464596c3ea2b48100707d24afec6334ac61cc46a1a1"} Nov 02 23:09:08.457700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.457646 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podStartSLOduration=1.45763238 podStartE2EDuration="1.45763238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:08.455058033 +0000 UTC m=+1262.884927912" watchObservedRunningTime="2026-11-02 23:09:08.45763238 +0000 UTC m=+1262.887502241" Nov 02 23:09:08.466553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:08.466522 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:09.004873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:09.004833 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:09.005311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:09.004888 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:09.699816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:09.699781 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:10.004586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.004502 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:10.004586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.004554 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:10.755202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.751012 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz"] Nov 02 23:09:10.761948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.761922 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:10.806662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.806632 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz"] Nov 02 23:09:10.856199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.853330 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:09:10.856199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.853497 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:09:10.905017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.904983 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:10.905238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.905049 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctkb\" (UniqueName: \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:10.905238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:10.905139 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.005179 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.005148 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:11.005375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.005244 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:11.005905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.005884 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.005992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.005977 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctkb\" (UniqueName: \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.006030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.006009 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.006628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.006605 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.008507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.008486 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.070853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.070814 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctkb\" (UniqueName: \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\") pod \"maas-api-revoke-sub-649c2fa9-4n4mz\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.076672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.076647 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:11.468859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:11.468830 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz"] Nov 02 23:09:11.469766 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:11.469743 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abcfa9a_1356_4212_910b_36fcb4087597.slice/crio-4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481 WatchSource:0}: Error finding container 4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481: Status 404 returned error can't find the container with id 4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481 Nov 02 23:09:12.005106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:12.005018 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:09:12.005106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:12.005088 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 23:09:12.397316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:12.396916 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" event={"ID":"9abcfa9a-1356-4212-910b-36fcb4087597","Type":"ContainerStarted","Data":"22b82c6d50a4e1b7a47abae2ffe587233b8abbd72515ec39880b875e49469323"} Nov 02 23:09:12.397316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:12.396981 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" event={"ID":"9abcfa9a-1356-4212-910b-36fcb4087597","Type":"ContainerStarted","Data":"4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481"} Nov 02 23:09:12.527411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:12.525611 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" podStartSLOduration=2.5255952390000003 podStartE2EDuration="2.525595239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:12.522664425 +0000 UTC m=+1266.952534288" watchObservedRunningTime="2026-11-02 23:09:12.525595239 +0000 UTC m=+1266.955465100" Nov 02 23:09:13.059197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.058394 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b"] Nov 02 23:09:13.118210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.114856 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a94d0f-md56b"] Nov 02 23:09:13.140561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.140508 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:13.144200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.142032 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:13.158209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.157448 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:09:13.996774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:13.996738 4494 generic.go:356] "Generic (PLEG): container finished" podID="9abcfa9a-1356-4212-910b-36fcb4087597" containerID="22b82c6d50a4e1b7a47abae2ffe587233b8abbd72515ec39880b875e49469323" exitCode=0 Nov 02 23:09:14.016336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.016299 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5e760d-f79a-470e-affc-f590970f9fa5" path="/var/lib/kubelet/pods/ed5e760d-f79a-470e-affc-f590970f9fa5/volumes" Nov 02 23:09:14.016770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.016742 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" event={"ID":"9abcfa9a-1356-4212-910b-36fcb4087597","Type":"ContainerDied","Data":"22b82c6d50a4e1b7a47abae2ffe587233b8abbd72515ec39880b875e49469323"} Nov 02 23:09:14.421278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.421249 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb"] Nov 02 23:09:14.425627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.425606 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.482493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.482462 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb"] Nov 02 23:09:14.563104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.563058 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.563305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.563280 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjp2s\" (UniqueName: \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.563402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.563385 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.664991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.664954 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjp2s\" (UniqueName: \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.665219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.665011 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.665219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.665201 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.665619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.665601 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.667597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.667574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.726953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.726878 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjp2s\" (UniqueName: \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\") pod \"maas-api-revoke-sub-4ed8334a-gc9bb\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.738713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.738689 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:14.949790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:14.949758 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:15.138520 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:15.138483 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d4a83f_b177_42d7_af43_c1f57e906089.slice/crio-e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68 WatchSource:0}: Error finding container e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68: Status 404 returned error can't find the container with id e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68 Nov 02 23:09:15.138520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.138490 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb"] Nov 02 23:09:15.203661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.203639 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:15.276478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.276450 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\") pod \"9abcfa9a-1356-4212-910b-36fcb4087597\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " Nov 02 23:09:15.276659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.276486 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\" (UniqueName: \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\") pod \"9abcfa9a-1356-4212-910b-36fcb4087597\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " Nov 02 23:09:15.276659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.276537 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\") pod \"9abcfa9a-1356-4212-910b-36fcb4087597\" (UID: \"9abcfa9a-1356-4212-910b-36fcb4087597\") " Nov 02 23:09:15.276870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.276846 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca" pod "9abcfa9a-1356-4212-910b-36fcb4087597" (UID: "9abcfa9a-1356-4212-910b-36fcb4087597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:15.277322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.277300 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:15.278720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.278700 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb" pod "9abcfa9a-1356-4212-910b-36fcb4087597" (UID: "9abcfa9a-1356-4212-910b-36fcb4087597"). InnerVolumeSpecName "kube-api-access-7ctkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:15.278806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.278733 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca" pod "9abcfa9a-1356-4212-910b-36fcb4087597" (UID: "9abcfa9a-1356-4212-910b-36fcb4087597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:15.378278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.378245 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctkb\" (UniqueName: \"kubernetes.io/projected/9abcfa9a-1356-4212-910b-36fcb4087597-kube-api-access-7ctkb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:15.378278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:15.378275 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abcfa9a-1356-4212-910b-36fcb4087597-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:16.034111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.034082 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" Nov 02 23:09:16.034499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.034105 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz" event={"ID":"9abcfa9a-1356-4212-910b-36fcb4087597","Type":"ContainerDied","Data":"4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481"} Nov 02 23:09:16.034499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.034302 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e690523f88953b76a27071177dad8681c0db3f5528df87ff9777bc91b5a6481" Nov 02 23:09:16.035921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.035900 4494 generic.go:356] "Generic (PLEG): container finished" podID="86d4a83f-b177-42d7-af43-c1f57e906089" containerID="3c8f8f55a9a7f2e915237bf460b4cef279cedfbb9ce70fb1076f5aa29ef044bc" exitCode=0 Nov 02 23:09:16.036011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.035956 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" event={"ID":"86d4a83f-b177-42d7-af43-c1f57e906089","Type":"ContainerDied","Data":"3c8f8f55a9a7f2e915237bf460b4cef279cedfbb9ce70fb1076f5aa29ef044bc"} Nov 02 23:09:16.036011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:16.035983 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" event={"ID":"86d4a83f-b177-42d7-af43-c1f57e906089","Type":"ContainerStarted","Data":"e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68"} Nov 02 23:09:17.186801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.186775 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:17.297839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.297741 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\") pod \"86d4a83f-b177-42d7-af43-c1f57e906089\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " Nov 02 23:09:17.297839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.297812 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\" (UniqueName: \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\") pod \"86d4a83f-b177-42d7-af43-c1f57e906089\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " Nov 02 23:09:17.298073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.297886 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\") pod \"86d4a83f-b177-42d7-af43-c1f57e906089\" (UID: \"86d4a83f-b177-42d7-af43-c1f57e906089\") " Nov 02 23:09:17.298141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.298117 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca" pod "86d4a83f-b177-42d7-af43-c1f57e906089" (UID: "86d4a83f-b177-42d7-af43-c1f57e906089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:17.298407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.298389 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:17.300126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.300099 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca" pod "86d4a83f-b177-42d7-af43-c1f57e906089" (UID: "86d4a83f-b177-42d7-af43-c1f57e906089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:17.300247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.300162 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s" pod "86d4a83f-b177-42d7-af43-c1f57e906089" (UID: "86d4a83f-b177-42d7-af43-c1f57e906089"). InnerVolumeSpecName "kube-api-access-sjp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:17.400098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.400067 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d4a83f-b177-42d7-af43-c1f57e906089-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:17.400098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.400094 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjp2s\" (UniqueName: \"kubernetes.io/projected/86d4a83f-b177-42d7-af43-c1f57e906089-kube-api-access-sjp2s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:17.466346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.466313 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:17.478990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:17.478968 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:17.559383 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:09:17.559288 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:09:18.048719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.048689 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" Nov 02 23:09:18.048889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.048734 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb" event={"ID":"86d4a83f-b177-42d7-af43-c1f57e906089","Type":"ContainerDied","Data":"e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68"} Nov 02 23:09:18.048889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.048791 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84bc82982f29b3de9aaacad9a0cc9c80f4cd871d1ac5eab17fb643641887b68" Nov 02 23:09:18.060756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.060736 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:09:18.307734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.307655 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:09:18.308274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308259 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9abcfa9a-1356-4212-910b-36fcb4087597" containerName="revoke-keys" Nov 02 23:09:18.308335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308276 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abcfa9a-1356-4212-910b-36fcb4087597" containerName="revoke-keys" Nov 02 23:09:18.308335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308309 4494 cpu_manager.go:414] "containerMap: removing container" podUID="86d4a83f-b177-42d7-af43-c1f57e906089" containerName="revoke-keys" Nov 02 23:09:18.308335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308314 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d4a83f-b177-42d7-af43-c1f57e906089" containerName="revoke-keys" Nov 02 23:09:18.308425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308408 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d4a83f-b177-42d7-af43-c1f57e906089" containerName="revoke-keys" Nov 02 23:09:18.308425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.308421 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abcfa9a-1356-4212-910b-36fcb4087597" containerName="revoke-keys" Nov 02 23:09:18.311903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.311884 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.399271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.399239 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:09:18.413105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413073 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413228 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvlmk\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413329 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413367 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413395 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413538 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413576 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413599 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.413685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.413647 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515389 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvlmk\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515443 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515463 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515479 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515566 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515608 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515636 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515750 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515817 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515844 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.515995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.515900 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.516095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.516075 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.516367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.516341 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.518272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.518249 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.518377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.518360 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.577904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.577828 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.578031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.577952 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvlmk\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:18.625511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:18.625465 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:19.058250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:19.056185 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:09:19.058810 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:19.058783 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a1b1f_85f2_4e8d_8e0f_0b606d6bced2.slice/crio-76ba9b7b3993bd42a8232ea3354a0c732daf78b8dcf4e129246aeb92267b0cb6 WatchSource:0}: Error finding container 76ba9b7b3993bd42a8232ea3354a0c732daf78b8dcf4e129246aeb92267b0cb6: Status 404 returned error can't find the container with id 76ba9b7b3993bd42a8232ea3354a0c732daf78b8dcf4e129246aeb92267b0cb6 Nov 02 23:09:19.061052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:19.061017 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:19.061132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:19.061101 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:19.061213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:19.061139 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:19.943209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:19.942776 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:20.073864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.073825 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" event={"ID":"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2","Type":"ContainerStarted","Data":"67ee3a5b4069c10a2b57b93634a24b1e947214adb0609d3d3f25a1ce480aa2cf"} Nov 02 23:09:20.074022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.073893 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" event={"ID":"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2","Type":"ContainerStarted","Data":"76ba9b7b3993bd42a8232ea3354a0c732daf78b8dcf4e129246aeb92267b0cb6"} Nov 02 23:09:20.257861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.257751 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podStartSLOduration=2.25773421 podStartE2EDuration="2.25773421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:20.256712266 +0000 UTC m=+1274.686582152" watchObservedRunningTime="2026-11-02 23:09:20.25773421 +0000 UTC m=+1274.687604078" Nov 02 23:09:20.626318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.626229 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:20.627724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.627702 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:20.627829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:20.627760 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:21.188555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:21.188512 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:21.627365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:21.626135 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:21.627365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:21.626240 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:22.626509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:22.626474 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:22.626997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:22.626533 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:23.626427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:23.626390 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:23.626627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:23.626459 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:24.626679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:24.626644 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:24.627083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:24.626699 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:24.891650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:24.891563 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:24.911690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:24.911661 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:09:25.167212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.167164 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn"] Nov 02 23:09:25.171468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.171447 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.228257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.228224 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:09:25.228564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.228539 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:09:25.231786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.231761 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn"] Nov 02 23:09:25.271570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.271534 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.271719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.271664 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.271719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.271700 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqzsk\" (UniqueName: \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.373598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.373560 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.373764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.373611 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqzsk\" (UniqueName: \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.373902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.373882 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.374283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.374262 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.376324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.376299 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.435507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.435437 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqzsk\" (UniqueName: \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\") pod \"maas-api-revoke-sub-d0251797-wk8sn\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.486787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.486749 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:25.626204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.626091 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:09:25.626204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.626156 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:09:25.929200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:25.915531 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn"] Nov 02 23:09:26.172785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:26.171666 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" event={"ID":"c7fdc364-a112-48f2-94eb-c005f3ebfe7f","Type":"ContainerStarted","Data":"e4d67dc7c9a6af84332a372c635b4ae82124578be3865b3172f60bd61e9148ae"} Nov 02 23:09:27.221199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.205249 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" event={"ID":"c7fdc364-a112-48f2-94eb-c005f3ebfe7f","Type":"ContainerStarted","Data":"0dbdf261fad0f6501d1ef3edcca656c74667132e843320fdc2d1f874131f2b4b"} Nov 02 23:09:27.277474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.277445 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:27.292197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.284992 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:27.297492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.297461 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:09:27.351074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.351000 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" podStartSLOduration=2.35097799 podStartE2EDuration="2.35097799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:27.348609322 +0000 UTC m=+1281.778479184" watchObservedRunningTime="2026-11-02 23:09:27.35097799 +0000 UTC m=+1281.780847851" Nov 02 23:09:27.999198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.996292 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:09:27.999198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:27.996359 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:28.109346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.109300 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:28.231627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.231589 4494 generic.go:356] "Generic (PLEG): container finished" podID="c7fdc364-a112-48f2-94eb-c005f3ebfe7f" containerID="0dbdf261fad0f6501d1ef3edcca656c74667132e843320fdc2d1f874131f2b4b" exitCode=0 Nov 02 23:09:28.241608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.241574 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" event={"ID":"c7fdc364-a112-48f2-94eb-c005f3ebfe7f","Type":"ContainerDied","Data":"0dbdf261fad0f6501d1ef3edcca656c74667132e843320fdc2d1f874131f2b4b"} Nov 02 23:09:28.440097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.424073 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm"] Nov 02 23:09:28.457013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.456981 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.577200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.552569 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm"] Nov 02 23:09:28.598557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.579674 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.598557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.595505 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswrd\" (UniqueName: \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.598557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.595623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.605927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.600258 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:28.719770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.719673 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswrd\" (UniqueName: \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.719770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.719741 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.720024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.719838 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.720620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.720599 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.726836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.726805 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.784941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.784909 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswrd\" (UniqueName: \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\") pod \"maas-api-revoke-sub-65f09147-6ncnm\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:28.926782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:28.926749 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:29.311343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.310354 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm"] Nov 02 23:09:29.317788 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:29.313015 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e5ee60_55dc_47cb_83df_97cc93c2607b.slice/crio-cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad WatchSource:0}: Error finding container cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad: Status 404 returned error can't find the container with id cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad Nov 02 23:09:29.386696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.386673 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:29.427677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.427641 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\") pod \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " Nov 02 23:09:29.427828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.427754 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\") pod \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " Nov 02 23:09:29.427828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.427785 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\" (UniqueName: \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\") pod \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\" (UID: \"c7fdc364-a112-48f2-94eb-c005f3ebfe7f\") " Nov 02 23:09:29.428135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.428112 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca" pod "c7fdc364-a112-48f2-94eb-c005f3ebfe7f" (UID: "c7fdc364-a112-48f2-94eb-c005f3ebfe7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:29.428421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.428405 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:29.429683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.429660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca" pod "c7fdc364-a112-48f2-94eb-c005f3ebfe7f" (UID: "c7fdc364-a112-48f2-94eb-c005f3ebfe7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:29.429952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.429933 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk" pod "c7fdc364-a112-48f2-94eb-c005f3ebfe7f" (UID: "c7fdc364-a112-48f2-94eb-c005f3ebfe7f"). InnerVolumeSpecName "kube-api-access-tqzsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:29.529776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.529706 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:29.529776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:29.529732 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqzsk\" (UniqueName: \"kubernetes.io/projected/c7fdc364-a112-48f2-94eb-c005f3ebfe7f-kube-api-access-tqzsk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:30.257453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.257426 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" Nov 02 23:09:30.257453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.257431 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn" event={"ID":"c7fdc364-a112-48f2-94eb-c005f3ebfe7f","Type":"ContainerDied","Data":"e4d67dc7c9a6af84332a372c635b4ae82124578be3865b3172f60bd61e9148ae"} Nov 02 23:09:30.257669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.257484 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d67dc7c9a6af84332a372c635b4ae82124578be3865b3172f60bd61e9148ae" Nov 02 23:09:30.259111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.259085 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" event={"ID":"b8e5ee60-55dc-47cb-83df-97cc93c2607b","Type":"ContainerStarted","Data":"461da9200514ce91463ef28c2c93d1b68d8729bd35522a01bc13bee6c3eb4dba"} Nov 02 23:09:30.259264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.259124 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" event={"ID":"b8e5ee60-55dc-47cb-83df-97cc93c2607b","Type":"ContainerStarted","Data":"cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad"} Nov 02 23:09:30.427447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.427398 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" podStartSLOduration=2.427382365 podStartE2EDuration="2.427382365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:30.425046612 +0000 UTC m=+1284.854916474" watchObservedRunningTime="2026-11-02 23:09:30.427382365 +0000 UTC m=+1284.857252226" Nov 02 23:09:30.558781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:30.558705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:31.048077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:31.048046 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld"] Nov 02 23:09:31.098492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:31.098463 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3feda2-wpfld"] Nov 02 23:09:31.266727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:31.266693 4494 generic.go:356] "Generic (PLEG): container finished" podID="b8e5ee60-55dc-47cb-83df-97cc93c2607b" containerID="461da9200514ce91463ef28c2c93d1b68d8729bd35522a01bc13bee6c3eb4dba" exitCode=0 Nov 02 23:09:31.266880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:31.266777 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" event={"ID":"b8e5ee60-55dc-47cb-83df-97cc93c2607b","Type":"ContainerDied","Data":"461da9200514ce91463ef28c2c93d1b68d8729bd35522a01bc13bee6c3eb4dba"} Nov 02 23:09:31.694095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:31.694056 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef9bb5c-a046-4106-9a53-4e0aceb2ce84" path="/var/lib/kubelet/pods/fef9bb5c-a046-4106-9a53-4e0aceb2ce84/volumes" Nov 02 23:09:32.421099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.421077 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:32.465786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.465756 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\") pod \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " Nov 02 23:09:32.465925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.465879 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\") pod \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " Nov 02 23:09:32.465925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.465919 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\" (UniqueName: \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\") pod \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\" (UID: \"b8e5ee60-55dc-47cb-83df-97cc93c2607b\") " Nov 02 23:09:32.466290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.466256 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca" pod "b8e5ee60-55dc-47cb-83df-97cc93c2607b" (UID: "b8e5ee60-55dc-47cb-83df-97cc93c2607b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:32.466428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.466413 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:32.467985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.467966 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca" pod "b8e5ee60-55dc-47cb-83df-97cc93c2607b" (UID: "b8e5ee60-55dc-47cb-83df-97cc93c2607b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:32.468136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.468111 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd" pod "b8e5ee60-55dc-47cb-83df-97cc93c2607b" (UID: "b8e5ee60-55dc-47cb-83df-97cc93c2607b"). InnerVolumeSpecName "kube-api-access-dswrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:32.568459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.568376 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswrd\" (UniqueName: \"kubernetes.io/projected/b8e5ee60-55dc-47cb-83df-97cc93c2607b-kube-api-access-dswrd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:32.568459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:32.568417 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8e5ee60-55dc-47cb-83df-97cc93c2607b-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:33.282369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:33.282340 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" Nov 02 23:09:33.282774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:33.282342 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm" event={"ID":"b8e5ee60-55dc-47cb-83df-97cc93c2607b","Type":"ContainerDied","Data":"cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad"} Nov 02 23:09:33.282774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:33.282463 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd9fc9e283b369949ae778da31757e2fcec1436d3b315bd8532acff631491ad" Nov 02 23:09:35.441330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:35.441293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:39.473712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.473659 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w"] Nov 02 23:09:39.481204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.475209 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b8e5ee60-55dc-47cb-83df-97cc93c2607b" containerName="revoke-keys" Nov 02 23:09:39.481421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.481391 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e5ee60-55dc-47cb-83df-97cc93c2607b" containerName="revoke-keys" Nov 02 23:09:39.481576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.481562 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c7fdc364-a112-48f2-94eb-c005f3ebfe7f" containerName="revoke-keys" Nov 02 23:09:39.481671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.481659 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fdc364-a112-48f2-94eb-c005f3ebfe7f" containerName="revoke-keys" Nov 02 23:09:39.481921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.481900 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e5ee60-55dc-47cb-83df-97cc93c2607b" containerName="revoke-keys" Nov 02 23:09:39.482063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.482049 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fdc364-a112-48f2-94eb-c005f3ebfe7f" containerName="revoke-keys" Nov 02 23:09:39.525824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.525790 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w"] Nov 02 23:09:39.526190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.526151 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.583131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.583097 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:09:39.651414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.610255 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.651414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.610319 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27nf\" (UniqueName: \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.651414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.610354 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.651414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.635848 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:09:39.796066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.795971 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.796340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.796311 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27nf\" (UniqueName: \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.796502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.796483 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.808743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.808714 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.819205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.809562 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:39.892212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:39.892159 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27nf\" (UniqueName: \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\") pod \"maas-api-revoke-sub-2836708d-bmj4w\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:40.162565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:40.162534 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:41.636162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.635645 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:09:41.636162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.635724 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:41.851571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.851511 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7"] Nov 02 23:09:41.925458 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:41.924955 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521e1ed0_51e4_47de_a993_29815ae80686.slice/crio-dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9 WatchSource:0}: Error finding container dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9: Status 404 returned error can't find the container with id dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9 Nov 02 23:09:41.929862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.929839 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w"] Nov 02 23:09:41.930220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.930202 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:41.939403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.939375 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7"] Nov 02 23:09:41.949740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.947091 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdf2\" (UniqueName: \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:41.949740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.947299 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:41.949740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:41.947396 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.048388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.048359 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.048538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.048430 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.048538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.048456 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdf2\" (UniqueName: \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.049828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.049803 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.051824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.051801 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.110491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.110462 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdf2\" (UniqueName: \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\") pod \"maas-api-revoke-sub-6057cb0a-wwlx7\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.281366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.281341 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:42.298267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.298231 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:42.646055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.645950 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" event={"ID":"521e1ed0-51e4-47de-a993-29815ae80686","Type":"ContainerStarted","Data":"770650fd40675826443aa8f1dc7d9e6b8112dbe7ee5a75666431837ae6b65c24"} Nov 02 23:09:42.646055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.646021 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" event={"ID":"521e1ed0-51e4-47de-a993-29815ae80686","Type":"ContainerStarted","Data":"dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9"} Nov 02 23:09:42.658209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.658165 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7"] Nov 02 23:09:42.658607 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:42.658586 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode657ebe0_e50a_47da_99a5_c7b208286dbf.slice/crio-25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41 WatchSource:0}: Error finding container 25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41: Status 404 returned error can't find the container with id 25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41 Nov 02 23:09:42.768987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:42.768944 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" podStartSLOduration=3.7689306 podStartE2EDuration="3.7689306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:42.768523246 +0000 UTC m=+1297.198393110" watchObservedRunningTime="2026-11-02 23:09:42.7689306 +0000 UTC m=+1297.198800461" Nov 02 23:09:43.653971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:43.653938 4494 generic.go:356] "Generic (PLEG): container finished" podID="521e1ed0-51e4-47de-a993-29815ae80686" containerID="770650fd40675826443aa8f1dc7d9e6b8112dbe7ee5a75666431837ae6b65c24" exitCode=0 Nov 02 23:09:43.654403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:43.654032 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" event={"ID":"521e1ed0-51e4-47de-a993-29815ae80686","Type":"ContainerDied","Data":"770650fd40675826443aa8f1dc7d9e6b8112dbe7ee5a75666431837ae6b65c24"} Nov 02 23:09:43.655761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:43.655735 4494 generic.go:356] "Generic (PLEG): container finished" podID="e657ebe0-e50a-47da-99a5-c7b208286dbf" containerID="3429c0bcfa26e1da50f4858ed57f0b550fc3483741562021a293f94d90139f9f" exitCode=0 Nov 02 23:09:43.655882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:43.655814 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" event={"ID":"e657ebe0-e50a-47da-99a5-c7b208286dbf","Type":"ContainerDied","Data":"3429c0bcfa26e1da50f4858ed57f0b550fc3483741562021a293f94d90139f9f"} Nov 02 23:09:43.655882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:43.655858 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" event={"ID":"e657ebe0-e50a-47da-99a5-c7b208286dbf","Type":"ContainerStarted","Data":"25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41"} Nov 02 23:09:44.425306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.425277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:44.831925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.831897 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:44.835245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.835220 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:44.883640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883610 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\") pod \"e657ebe0-e50a-47da-99a5-c7b208286dbf\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " Nov 02 23:09:44.883743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883680 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\") pod \"521e1ed0-51e4-47de-a993-29815ae80686\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " Nov 02 23:09:44.883743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883735 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\") pod \"e657ebe0-e50a-47da-99a5-c7b208286dbf\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " Nov 02 23:09:44.883818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883761 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\" (UniqueName: \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\") pod \"521e1ed0-51e4-47de-a993-29815ae80686\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " Nov 02 23:09:44.883818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883803 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\" (UniqueName: \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\") pod \"e657ebe0-e50a-47da-99a5-c7b208286dbf\" (UID: \"e657ebe0-e50a-47da-99a5-c7b208286dbf\") " Nov 02 23:09:44.883977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.883957 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\") pod \"521e1ed0-51e4-47de-a993-29815ae80686\" (UID: \"521e1ed0-51e4-47de-a993-29815ae80686\") " Nov 02 23:09:44.884228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.884200 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca" pod "e657ebe0-e50a-47da-99a5-c7b208286dbf" (UID: "e657ebe0-e50a-47da-99a5-c7b208286dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:44.884414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.884392 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca" pod "521e1ed0-51e4-47de-a993-29815ae80686" (UID: "521e1ed0-51e4-47de-a993-29815ae80686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:09:44.884606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.884589 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:44.884664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.884610 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:44.886524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.886505 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca" pod "521e1ed0-51e4-47de-a993-29815ae80686" (UID: "521e1ed0-51e4-47de-a993-29815ae80686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:44.886632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.886531 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf" pod "521e1ed0-51e4-47de-a993-29815ae80686" (UID: "521e1ed0-51e4-47de-a993-29815ae80686"). InnerVolumeSpecName "kube-api-access-f27nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:44.886927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.886908 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2" pod "e657ebe0-e50a-47da-99a5-c7b208286dbf" (UID: "e657ebe0-e50a-47da-99a5-c7b208286dbf"). InnerVolumeSpecName "kube-api-access-ckdf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:09:44.886978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.886908 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca" pod "e657ebe0-e50a-47da-99a5-c7b208286dbf" (UID: "e657ebe0-e50a-47da-99a5-c7b208286dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:09:44.985671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.985596 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e657ebe0-e50a-47da-99a5-c7b208286dbf-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:44.985671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.985622 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521e1ed0-51e4-47de-a993-29815ae80686-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:44.985671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.985632 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f27nf\" (UniqueName: \"kubernetes.io/projected/521e1ed0-51e4-47de-a993-29815ae80686-kube-api-access-f27nf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:44.985671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:44.985642 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdf2\" (UniqueName: \"kubernetes.io/projected/e657ebe0-e50a-47da-99a5-c7b208286dbf-kube-api-access-ckdf2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:09:45.670265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.670236 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" Nov 02 23:09:45.670265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.670260 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w" event={"ID":"521e1ed0-51e4-47de-a993-29815ae80686","Type":"ContainerDied","Data":"dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9"} Nov 02 23:09:45.670501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.670310 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8b27b69eb26eae5d2c84e66bf0b6150f1096ef958ae7372210e21e2f211dc9" Nov 02 23:09:45.671714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.671695 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" Nov 02 23:09:45.671833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.671745 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7" event={"ID":"e657ebe0-e50a-47da-99a5-c7b208286dbf","Type":"ContainerDied","Data":"25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41"} Nov 02 23:09:45.671833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:45.671785 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a0bce2b7a9387936f2a2d115028adb4279253f9acad6c91e2eaf0b0c801e41" Nov 02 23:09:47.237231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:47.237151 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:47.563276 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:09:47.563146 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:09:50.062876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:50.062837 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:09:51.136883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.136834 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:09:51.138678 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.138655 4494 cpu_manager.go:414] "containerMap: removing container" podUID="521e1ed0-51e4-47de-a993-29815ae80686" containerName="revoke-keys" Nov 02 23:09:51.138875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.138859 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521e1ed0-51e4-47de-a993-29815ae80686" containerName="revoke-keys" Nov 02 23:09:51.139011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.138998 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e657ebe0-e50a-47da-99a5-c7b208286dbf" containerName="revoke-keys" Nov 02 23:09:51.139122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.139109 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e657ebe0-e50a-47da-99a5-c7b208286dbf" containerName="revoke-keys" Nov 02 23:09:51.139367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.139351 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="521e1ed0-51e4-47de-a993-29815ae80686" containerName="revoke-keys" Nov 02 23:09:51.139502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.139490 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e657ebe0-e50a-47da-99a5-c7b208286dbf" containerName="revoke-keys" Nov 02 23:09:51.145224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.145203 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.202371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.201568 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-96031a-openshift-default-dockercfg-fx8gh\"" Nov 02 23:09:51.315011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.314959 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.315291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.315264 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.315487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.315466 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7j9w\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.315663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.315646 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.315809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.315793 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.315933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.315917 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.316060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.316045 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.316291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.316271 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.316454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.316437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.393258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.387229 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:09:51.441199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.437947 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.454202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.450263 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.464200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.460285 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7j9w\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.478362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.478326 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.482200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.478597 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.490296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.490255 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.496283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.496254 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.522320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.505344 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.532327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.532292 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.532633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.481051 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.532774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.500367 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.532882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.499500 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.532981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.499734 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.533406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.533384 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.614196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.551471 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.614196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.602748 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.720202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.715823 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:51.819394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:51.819357 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7j9w\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:52.089257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:52.088454 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:52.556280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:52.555261 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 23:09:53.343959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:53.343909 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:09:53.382825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:53.382141 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:53.382825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:53.382285 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:53.382825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:53.382338 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:09:53.514731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:53.514632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:54.027967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.027931 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" event={"ID":"c0f44320-2b49-42dd-845e-b4a06cc3a0b4","Type":"ContainerStarted","Data":"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1"} Nov 02 23:09:54.028402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.027995 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" event={"ID":"c0f44320-2b49-42dd-845e-b4a06cc3a0b4","Type":"ContainerStarted","Data":"9da022af49e9c979fc269dfa5fae5fdd7d3b20d00245a59d39a47b2e8717a698"} Nov 02 23:09:54.089281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.089245 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:09:54.090835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.090807 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:54.090955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.090862 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:54.266140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:54.266082 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podStartSLOduration=3.266061776 podStartE2EDuration="3.266061776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:54.209771566 +0000 UTC m=+1308.639641429" watchObservedRunningTime="2026-11-02 23:09:54.266061776 +0000 UTC m=+1308.695931643" Nov 02 23:09:55.089344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:55.089309 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:55.089744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:55.089366 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:56.089641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.089605 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:56.090147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.089655 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:56.365563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.365466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:56.586286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.586251 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd"] Nov 02 23:09:56.590608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.590583 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.668444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.668302 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.668738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.668616 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.668923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.668903 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnp5\" (UniqueName: \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.693196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.687568 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd"] Nov 02 23:09:56.700558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.700523 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:09:56.700943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.700922 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:09:56.784296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.779718 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.784296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.779814 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnp5\" (UniqueName: \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.784296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.780082 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.784296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.781015 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.790313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.790282 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.860394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.860322 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnp5\" (UniqueName: \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\") pod \"maas-api-revoke-sub-e5ff051d-z8pwd\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:56.907143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:56.907105 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:09:57.089795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.089757 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:57.090244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.089817 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:57.157796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.157726 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz"] Nov 02 23:09:57.163417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.163392 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.185995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.185966 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx6xb\" (UniqueName: \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.186121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.186008 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.186226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.186206 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.205948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.205914 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz"] Nov 02 23:09:57.287471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.287436 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx6xb\" (UniqueName: \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.287649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.287481 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.287649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.287610 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.288250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.288230 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.290160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.290139 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.317839 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:57.317808 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb0b90b_3d76_499b_991c_5762f35b0945.slice/crio-0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478 WatchSource:0}: Error finding container 0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478: Status 404 returned error can't find the container with id 0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478 Nov 02 23:09:57.317839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.317822 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd"] Nov 02 23:09:57.366090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.366051 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx6xb\" (UniqueName: \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\") pod \"maas-api-revoke-sub-b5ce091a-q59jz\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.481583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.481258 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:09:57.957313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:57.957260 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz"] Nov 02 23:09:57.963435 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:09:57.963401 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048069e_453c_4967_b4b9_0e688b705f6b.slice/crio-9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb WatchSource:0}: Error finding container 9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb: Status 404 returned error can't find the container with id 9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb Nov 02 23:09:58.084115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.083992 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" event={"ID":"4048069e-453c-4967-b4b9-0e688b705f6b","Type":"ContainerStarted","Data":"9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb"} Nov 02 23:09:58.085980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.085768 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" event={"ID":"8eb0b90b-3d76-499b-991c-5762f35b0945","Type":"ContainerStarted","Data":"b2d69ef365d30064fb256191855f417d7f928d3c25811741fdfecdbdf41d995f"} Nov 02 23:09:58.085980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.085816 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" event={"ID":"8eb0b90b-3d76-499b-991c-5762f35b0945","Type":"ContainerStarted","Data":"0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478"} Nov 02 23:09:58.089918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.089847 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:58.089918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.089900 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:58.208532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.208310 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" podStartSLOduration=2.20829182 podStartE2EDuration="2.20829182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:58.205127548 +0000 UTC m=+1312.634997410" watchObservedRunningTime="2026-11-02 23:09:58.20829182 +0000 UTC m=+1312.638161682" Nov 02 23:09:58.798165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:58.798126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:09:59.089300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.089216 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:09:59.089300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.089280 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:09:59.096078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.096044 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" event={"ID":"4048069e-453c-4967-b4b9-0e688b705f6b","Type":"ContainerStarted","Data":"bf41f197bd6549dedd7438a7f188916f2b966a323772f25a2dc99a9efc95a1cb"} Nov 02 23:09:59.097777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.097741 4494 generic.go:356] "Generic (PLEG): container finished" podID="8eb0b90b-3d76-499b-991c-5762f35b0945" containerID="b2d69ef365d30064fb256191855f417d7f928d3c25811741fdfecdbdf41d995f" exitCode=0 Nov 02 23:09:59.097851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.097817 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" event={"ID":"8eb0b90b-3d76-499b-991c-5762f35b0945","Type":"ContainerDied","Data":"b2d69ef365d30064fb256191855f417d7f928d3c25811741fdfecdbdf41d995f"} Nov 02 23:09:59.219355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:09:59.219301 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" podStartSLOduration=2.219285231 podStartE2EDuration="2.219285231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:59.21724202 +0000 UTC m=+1313.647111900" watchObservedRunningTime="2026-11-02 23:09:59.219285231 +0000 UTC m=+1313.649155092" Nov 02 23:10:00.090260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.090223 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 23:10:00.090450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.090288 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 23:10:00.106872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.106835 4494 generic.go:356] "Generic (PLEG): container finished" podID="4048069e-453c-4967-b4b9-0e688b705f6b" containerID="bf41f197bd6549dedd7438a7f188916f2b966a323772f25a2dc99a9efc95a1cb" exitCode=0 Nov 02 23:10:00.107389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.106920 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" event={"ID":"4048069e-453c-4967-b4b9-0e688b705f6b","Type":"ContainerDied","Data":"bf41f197bd6549dedd7438a7f188916f2b966a323772f25a2dc99a9efc95a1cb"} Nov 02 23:10:00.279965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.279938 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:10:00.358523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.358416 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\") pod \"8eb0b90b-3d76-499b-991c-5762f35b0945\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " Nov 02 23:10:00.358523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.358460 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\" (UniqueName: \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\") pod \"8eb0b90b-3d76-499b-991c-5762f35b0945\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " Nov 02 23:10:00.358777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.358638 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\") pod \"8eb0b90b-3d76-499b-991c-5762f35b0945\" (UID: \"8eb0b90b-3d76-499b-991c-5762f35b0945\") " Nov 02 23:10:00.358885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.358860 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca" pod "8eb0b90b-3d76-499b-991c-5762f35b0945" (UID: "8eb0b90b-3d76-499b-991c-5762f35b0945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:00.359210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.359165 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:00.360778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.360754 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5" pod "8eb0b90b-3d76-499b-991c-5762f35b0945" (UID: "8eb0b90b-3d76-499b-991c-5762f35b0945"). InnerVolumeSpecName "kube-api-access-wnnp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:00.361265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.361231 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca" pod "8eb0b90b-3d76-499b-991c-5762f35b0945" (UID: "8eb0b90b-3d76-499b-991c-5762f35b0945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:00.461024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.460982 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnp5\" (UniqueName: \"kubernetes.io/projected/8eb0b90b-3d76-499b-991c-5762f35b0945-kube-api-access-wnnp5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:00.461024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.461017 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb0b90b-3d76-499b-991c-5762f35b0945-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:00.922342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.922308 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6"] Nov 02 23:10:00.922913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.922900 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8eb0b90b-3d76-499b-991c-5762f35b0945" containerName="revoke-keys" Nov 02 23:10:00.922958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.922914 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb0b90b-3d76-499b-991c-5762f35b0945" containerName="revoke-keys" Nov 02 23:10:00.923020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.923011 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb0b90b-3d76-499b-991c-5762f35b0945" containerName="revoke-keys" Nov 02 23:10:00.926720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.926705 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:00.986081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:00.986048 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6"] Nov 02 23:10:01.057405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.057368 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77"] Nov 02 23:10:01.069658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.069630 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8n5v\" (UniqueName: \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.069809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.069670 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.069809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.069788 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.095665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.095635 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a837ea23-c2k77"] Nov 02 23:10:01.118845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.118816 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" Nov 02 23:10:01.119360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.118846 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd" event={"ID":"8eb0b90b-3d76-499b-991c-5762f35b0945","Type":"ContainerDied","Data":"0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478"} Nov 02 23:10:01.119360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.118916 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0301ee680fa4f2801758467e387bc05f0709887d80ea27e1b8bc321bdf4a8478" Nov 02 23:10:01.171751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.171715 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.172229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.172204 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8n5v\" (UniqueName: \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.172431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.172382 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.173381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.173352 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.174763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.174739 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.235293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.235260 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8n5v\" (UniqueName: \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\") pod \"maas-api-revoke-sub-cbe483e7-nq4p6\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.240970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.240940 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:01.283279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.283241 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:10:01.346789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.346671 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:10:01.347860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.347812 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:10:01.349124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.349066 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:10:01.375524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.375484 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\" (UniqueName: \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\") pod \"4048069e-453c-4967-b4b9-0e688b705f6b\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " Nov 02 23:10:01.375746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.375723 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\") pod \"4048069e-453c-4967-b4b9-0e688b705f6b\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " Nov 02 23:10:01.375846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.375769 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\") pod \"4048069e-453c-4967-b4b9-0e688b705f6b\" (UID: \"4048069e-453c-4967-b4b9-0e688b705f6b\") " Nov 02 23:10:01.376146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.376107 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca" pod "4048069e-453c-4967-b4b9-0e688b705f6b" (UID: "4048069e-453c-4967-b4b9-0e688b705f6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:01.376361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.376318 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:01.377926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.377899 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb" pod "4048069e-453c-4967-b4b9-0e688b705f6b" (UID: "4048069e-453c-4967-b4b9-0e688b705f6b"). InnerVolumeSpecName "kube-api-access-cx6xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:01.378034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.377911 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca" pod "4048069e-453c-4967-b4b9-0e688b705f6b" (UID: "4048069e-453c-4967-b4b9-0e688b705f6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:01.478238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.478116 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4048069e-453c-4967-b4b9-0e688b705f6b-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:01.478238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.478145 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx6xb\" (UniqueName: \"kubernetes.io/projected/4048069e-453c-4967-b4b9-0e688b705f6b-kube-api-access-cx6xb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:01.647356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.647296 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6"] Nov 02 23:10:01.648617 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:01.648578 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead488c4_a1f7_4a49_83b9_19c9009d273e.slice/crio-091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed WatchSource:0}: Error finding container 091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed: Status 404 returned error can't find the container with id 091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed Nov 02 23:10:01.695603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:01.695564 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f51b62-6a60-4491-9d83-1ee1001d4903" path="/var/lib/kubelet/pods/28f51b62-6a60-4491-9d83-1ee1001d4903/volumes" Nov 02 23:10:02.050158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.050117 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h"] Nov 02 23:10:02.053422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.053399 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca737964-h7x2h"] Nov 02 23:10:02.126633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.126535 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" Nov 02 23:10:02.126633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.126547 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz" event={"ID":"4048069e-453c-4967-b4b9-0e688b705f6b","Type":"ContainerDied","Data":"9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb"} Nov 02 23:10:02.126633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.126594 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b844256875a5d74f649e3d82ec63407b5016219b6400d6afff5feeceaae09cb" Nov 02 23:10:02.128336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.128305 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" event={"ID":"ead488c4-a1f7-4a49-83b9-19c9009d273e","Type":"ContainerStarted","Data":"6ce3e462c117db536169972444aa8cbf6096ba5dd6bae755029307990f77b72c"} Nov 02 23:10:02.128433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.128355 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" event={"ID":"ead488c4-a1f7-4a49-83b9-19c9009d273e","Type":"ContainerStarted","Data":"091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed"} Nov 02 23:10:02.248532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.248476 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" podStartSLOduration=2.248461962 podStartE2EDuration="2.248461962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:02.246014694 +0000 UTC m=+1316.675884557" watchObservedRunningTime="2026-11-02 23:10:02.248461962 +0000 UTC m=+1316.678331822" Nov 02 23:10:02.688617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:02.688577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:03.137258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.137223 4494 generic.go:356] "Generic (PLEG): container finished" podID="ead488c4-a1f7-4a49-83b9-19c9009d273e" containerID="6ce3e462c117db536169972444aa8cbf6096ba5dd6bae755029307990f77b72c" exitCode=0 Nov 02 23:10:03.137799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.137318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" event={"ID":"ead488c4-a1f7-4a49-83b9-19c9009d273e","Type":"ContainerDied","Data":"6ce3e462c117db536169972444aa8cbf6096ba5dd6bae755029307990f77b72c"} Nov 02 23:10:03.661858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.661828 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:10:03.662530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.662511 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4048069e-453c-4967-b4b9-0e688b705f6b" containerName="revoke-keys" Nov 02 23:10:03.662658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.662531 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048069e-453c-4967-b4b9-0e688b705f6b" containerName="revoke-keys" Nov 02 23:10:03.662713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.662687 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048069e-453c-4967-b4b9-0e688b705f6b" containerName="revoke-keys" Nov 02 23:10:03.666266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.666246 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.699846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.699813 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d96c498-7c17-4b2f-b67a-ebee25bc5696" path="/var/lib/kubelet/pods/0d96c498-7c17-4b2f-b67a-ebee25bc5696/volumes" Nov 02 23:10:03.808078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808047 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808200 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808239 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808265 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808292 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grg69\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808431 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808452 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808494 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.808696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.808673 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.843969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.843938 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:03.891196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.879113 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915193 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915265 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915380 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915428 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915474 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915506 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grg69\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915572 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915601 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.916003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.915667 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.928205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.923920 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.954060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.939125 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.954060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.939482 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.954060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.939763 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.954060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.940393 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.954060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.940891 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.971692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.971646 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:03.995891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:03.995838 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:04.003583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.003553 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grg69\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:04.281363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.281125 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:04.292889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.292867 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:04.421234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.421201 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\") pod \"ead488c4-a1f7-4a49-83b9-19c9009d273e\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " Nov 02 23:10:04.421443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.421427 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\") pod \"ead488c4-a1f7-4a49-83b9-19c9009d273e\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " Nov 02 23:10:04.421550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.421537 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\" (UniqueName: \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\") pod \"ead488c4-a1f7-4a49-83b9-19c9009d273e\" (UID: \"ead488c4-a1f7-4a49-83b9-19c9009d273e\") " Nov 02 23:10:04.421940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.421673 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca" pod "ead488c4-a1f7-4a49-83b9-19c9009d273e" (UID: "ead488c4-a1f7-4a49-83b9-19c9009d273e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:04.422216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.422080 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:04.423738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.423713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v" pod "ead488c4-a1f7-4a49-83b9-19c9009d273e" (UID: "ead488c4-a1f7-4a49-83b9-19c9009d273e"). InnerVolumeSpecName "kube-api-access-m8n5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:04.424469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.424445 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca" pod "ead488c4-a1f7-4a49-83b9-19c9009d273e" (UID: "ead488c4-a1f7-4a49-83b9-19c9009d273e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:04.523234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.523133 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead488c4-a1f7-4a49-83b9-19c9009d273e-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:04.523234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.523182 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8n5v\" (UniqueName: \"kubernetes.io/projected/ead488c4-a1f7-4a49-83b9-19c9009d273e-kube-api-access-m8n5v\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:04.711756 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:04.711720 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f278669_ff51_4041_bc73_64448271df55.slice/crio-1f84de758dff18384db4f20f3803a165f5641cb1eca75fefa7a8258bb4f02262 WatchSource:0}: Error finding container 1f84de758dff18384db4f20f3803a165f5641cb1eca75fefa7a8258bb4f02262: Status 404 returned error can't find the container with id 1f84de758dff18384db4f20f3803a165f5641cb1eca75fefa7a8258bb4f02262 Nov 02 23:10:04.712296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.712275 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:10:04.714335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.714303 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:04.714423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.714369 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:04.714423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:04.714399 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.133646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.133614 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:10:05.134230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.134216 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ead488c4-a1f7-4a49-83b9-19c9009d273e" containerName="revoke-keys" Nov 02 23:10:05.134281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.134232 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead488c4-a1f7-4a49-83b9-19c9009d273e" containerName="revoke-keys" Nov 02 23:10:05.134347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.134339 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead488c4-a1f7-4a49-83b9-19c9009d273e" containerName="revoke-keys" Nov 02 23:10:05.137850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.137828 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.155931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.155907 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" Nov 02 23:10:05.156077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.155949 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6" event={"ID":"ead488c4-a1f7-4a49-83b9-19c9009d273e","Type":"ContainerDied","Data":"091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed"} Nov 02 23:10:05.156077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.156005 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091d1be51c7effe1a38be68d6c20c35acbfa3d432a94ed9376f9f4f8af16a1ed" Nov 02 23:10:05.157877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.157843 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" event={"ID":"8f278669-ff51-4041-bc73-64448271df55","Type":"ContainerStarted","Data":"50008c62fb4ac512cbd0dbb1b728831cf50d2a25778796e37f9d405172b8447f"} Nov 02 23:10:05.158006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.157898 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" event={"ID":"8f278669-ff51-4041-bc73-64448271df55","Type":"ContainerStarted","Data":"1f84de758dff18384db4f20f3803a165f5641cb1eca75fefa7a8258bb4f02262"} Nov 02 23:10:05.226435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.226313 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:10:05.231707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.231674 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.231826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.231812 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.231863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.231840 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.231919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.231887 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.232018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.232004 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.232063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.232027 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.232063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.232054 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s8h\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.232150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.232074 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.232218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.232153 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.284221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.282401 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:05.284752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.284375 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:05.284752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.284489 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345256 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345402 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345439 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345525 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345656 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345687 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345746 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s8h\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345780 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.345813 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.351656 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.352660 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.358133 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.358448 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.359959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.359255 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.362951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.362923 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.364882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.364853 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.381722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.381674 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podStartSLOduration=2.381661585 podStartE2EDuration="2.381661585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:05.379583484 +0000 UTC m=+1319.809453353" watchObservedRunningTime="2026-11-02 23:10:05.381661585 +0000 UTC m=+1319.811531445" Nov 02 23:10:05.415498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.415463 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.425762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.425736 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s8h\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\") pod \"e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.457393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.456316 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:05.893092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.893063 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:10:05.894152 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:05.894127 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64f1ca3_5ad8_4e42_b1da_134c377aacdc.slice/crio-13dd18d73f89a0b3ae76eb5cf64b4535409698226b6b1c82a033d1a00ccc3920 WatchSource:0}: Error finding container 13dd18d73f89a0b3ae76eb5cf64b4535409698226b6b1c82a033d1a00ccc3920: Status 404 returned error can't find the container with id 13dd18d73f89a0b3ae76eb5cf64b4535409698226b6b1c82a033d1a00ccc3920 Nov 02 23:10:05.896990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.896950 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.897083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.897044 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:05.897121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:05.897088 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:06.247872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.247819 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" event={"ID":"e64f1ca3-5ad8-4e42-b1da-134c377aacdc","Type":"ContainerStarted","Data":"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d"} Nov 02 23:10:06.265201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.248192 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" event={"ID":"e64f1ca3-5ad8-4e42-b1da-134c377aacdc","Type":"ContainerStarted","Data":"13dd18d73f89a0b3ae76eb5cf64b4535409698226b6b1c82a033d1a00ccc3920"} Nov 02 23:10:06.296206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.292082 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:06.296206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.292138 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:06.385198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.384382 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podStartSLOduration=1.384358355 podStartE2EDuration="1.384358355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:06.372190356 +0000 UTC m=+1320.802060217" watchObservedRunningTime="2026-11-02 23:10:06.384358355 +0000 UTC m=+1320.814228219" Nov 02 23:10:06.458198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:06.457381 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:07.292820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:07.285296 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.292820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:07.285381 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:07.465207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:07.465146 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:07.465898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:07.465858 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:08.058328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.058270 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:08.287490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.282566 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.287490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.282641 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:08.459609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.459571 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:08.459912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.459880 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:08.545152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:08.545108 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.061344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.061300 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:09.062940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.062917 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:10:09.276200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.265057 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk"] Nov 02 23:10:09.316257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.303100 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:09.488256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.451695 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:09.488256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.451755 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:09.488256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.451861 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:10:09.556744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.556479 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:10:09.653259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.619961 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk"] Nov 02 23:10:09.689529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.689482 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:09.689795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.689760 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:09.759241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.755246 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:09.765508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.765477 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8xbp\" (UniqueName: \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:09.782349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.782284 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:09.917270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:09.908019 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8xbp\" (UniqueName: \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.053239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.048923 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.053239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.049109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.053239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.050036 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.053239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.048580 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 02 23:10:10.122497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.122466 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.310260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.299098 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.310582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.310542 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:10.480128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.478335 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:10.480128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.478395 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:10.533215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.507876 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8xbp\" (UniqueName: \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\") pod \"maas-api-revoke-sub-77bc78f0-k9qkk\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.584397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.584302 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:10.939330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:10.939284 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 02 23:10:11.289262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.286073 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.289262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.286136 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:11.335316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.335274 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:10:11.335566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.335535 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.677254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.665376 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:10:11.677254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.665434 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:11.827915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.818403 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:11.827915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:11.818463 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:12.354198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.349017 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.354198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.349083 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:12.426279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.419126 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk"] Nov 02 23:10:12.528454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.520347 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:12.528454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.520437 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:12.701212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.690280 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:12.775315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.775246 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds"] Nov 02 23:10:12.796869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.796830 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:12.857257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.853771 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:12.876869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:12.865050 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds"] Nov 02 23:10:13.092817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.092760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.093131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.093106 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.093398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.093374 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5s45\" (UniqueName: \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.187850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.187813 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" event={"ID":"c7292e94-19f0-4336-a312-39ee18b66d5a","Type":"ContainerStarted","Data":"763d51a4709b529857d5b0edf639a411bcf2c3b268292323327531210d70db72"} Nov 02 23:10:13.194468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.194437 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5s45\" (UniqueName: \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.194561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.194540 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.194629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.194606 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.195319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.195300 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.260230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.260195 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.280309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.276024 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5s45\" (UniqueName: \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\") pod \"maas-api-revoke-sub-2975ef91-5tlds\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.286227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.285963 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:13.286227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.286038 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:13.386949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.386904 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:13.404542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.404512 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b"] Nov 02 23:10:13.411440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.411417 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.463198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.458739 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:13.463198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.458813 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:13.496209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.495472 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b"] Nov 02 23:10:13.498732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.498709 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.498848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.498814 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5dh\" (UniqueName: \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.499070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.498976 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.602214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.601863 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.602214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.601914 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5dh\" (UniqueName: \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.602214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.601963 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.602718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.602693 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.604869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.604844 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.682068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.681939 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5dh\" (UniqueName: \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\") pod \"maas-api-revoke-sub-0ee1a93d-nkg8b\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.740251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.736405 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:13.843194 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:13.830091 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4f0a6f_4e46_4cdc_bf52_a70e9e1877be.slice/crio-37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1 WatchSource:0}: Error finding container 37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1: Status 404 returned error can't find the container with id 37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1 Nov 02 23:10:13.843194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:13.830281 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds"] Nov 02 23:10:14.198407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.198376 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" event={"ID":"c7292e94-19f0-4336-a312-39ee18b66d5a","Type":"ContainerStarted","Data":"836fcd986525d47eeb6704881cfceadfbfb90242cb1bafc788aa396fb6aa911a"} Nov 02 23:10:14.200046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.200025 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" event={"ID":"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be","Type":"ContainerStarted","Data":"37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1"} Nov 02 23:10:14.222026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.222001 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b"] Nov 02 23:10:14.282697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.282666 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:14.282818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.282721 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:14.477440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.471257 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:14.477440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:14.471310 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:15.210208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.210079 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" event={"ID":"307c5897-ae78-484c-8ddc-801d53a138fe","Type":"ContainerStarted","Data":"680da2f544d94ced14f752c7f875532a5dc3b4fcff428e5d4fd472b87fbfc064"} Nov 02 23:10:15.210208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.210206 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" event={"ID":"307c5897-ae78-484c-8ddc-801d53a138fe","Type":"ContainerStarted","Data":"4b82cd691bd079787cdd5c484832de35aa23a1abbd279a48f3c9ba626a969dd9"} Nov 02 23:10:15.212186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.212133 4494 generic.go:356] "Generic (PLEG): container finished" podID="c7292e94-19f0-4336-a312-39ee18b66d5a" containerID="836fcd986525d47eeb6704881cfceadfbfb90242cb1bafc788aa396fb6aa911a" exitCode=0 Nov 02 23:10:15.212313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.212253 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" event={"ID":"c7292e94-19f0-4336-a312-39ee18b66d5a","Type":"ContainerDied","Data":"836fcd986525d47eeb6704881cfceadfbfb90242cb1bafc788aa396fb6aa911a"} Nov 02 23:10:15.214705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.214559 4494 generic.go:356] "Generic (PLEG): container finished" podID="9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" containerID="0c968dc5f2808eac23e810417db50ebb77755b89e22bf64761b0ec4fe6760117" exitCode=0 Nov 02 23:10:15.214705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.214672 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" event={"ID":"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be","Type":"ContainerDied","Data":"0c968dc5f2808eac23e810417db50ebb77755b89e22bf64761b0ec4fe6760117"} Nov 02 23:10:15.283214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.282875 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 23:10:15.283214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.282946 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 23:10:15.391295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.390110 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" podStartSLOduration=2.390095331 podStartE2EDuration="2.390095331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:15.387493696 +0000 UTC m=+1329.817363535" watchObservedRunningTime="2026-11-02 23:10:15.390095331 +0000 UTC m=+1329.819965192" Nov 02 23:10:15.461610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.461562 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:15.461776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.461627 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:15.472048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.472023 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:15.591494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.591462 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\") pod \"c7292e94-19f0-4336-a312-39ee18b66d5a\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " Nov 02 23:10:15.591667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.591648 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\" (UniqueName: \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\") pod \"c7292e94-19f0-4336-a312-39ee18b66d5a\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " Nov 02 23:10:15.591866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.591847 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\") pod \"c7292e94-19f0-4336-a312-39ee18b66d5a\" (UID: \"c7292e94-19f0-4336-a312-39ee18b66d5a\") " Nov 02 23:10:15.591910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.591890 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca" pod "c7292e94-19f0-4336-a312-39ee18b66d5a" (UID: "c7292e94-19f0-4336-a312-39ee18b66d5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:15.592345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.592319 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:15.593870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.593844 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp" pod "c7292e94-19f0-4336-a312-39ee18b66d5a" (UID: "c7292e94-19f0-4336-a312-39ee18b66d5a"). InnerVolumeSpecName "kube-api-access-w8xbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:15.593990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.593970 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca" pod "c7292e94-19f0-4336-a312-39ee18b66d5a" (UID: "c7292e94-19f0-4336-a312-39ee18b66d5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:15.693685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.693649 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8xbp\" (UniqueName: \"kubernetes.io/projected/c7292e94-19f0-4336-a312-39ee18b66d5a-kube-api-access-w8xbp\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:15.693685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:15.693679 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7292e94-19f0-4336-a312-39ee18b66d5a-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:16.224306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.224267 4494 generic.go:356] "Generic (PLEG): container finished" podID="307c5897-ae78-484c-8ddc-801d53a138fe" containerID="680da2f544d94ced14f752c7f875532a5dc3b4fcff428e5d4fd472b87fbfc064" exitCode=0 Nov 02 23:10:16.224745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.224350 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" event={"ID":"307c5897-ae78-484c-8ddc-801d53a138fe","Type":"ContainerDied","Data":"680da2f544d94ced14f752c7f875532a5dc3b4fcff428e5d4fd472b87fbfc064"} Nov 02 23:10:16.225756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.225727 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" event={"ID":"c7292e94-19f0-4336-a312-39ee18b66d5a","Type":"ContainerDied","Data":"763d51a4709b529857d5b0edf639a411bcf2c3b268292323327531210d70db72"} Nov 02 23:10:16.225756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.225754 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk" Nov 02 23:10:16.225889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.225776 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763d51a4709b529857d5b0edf639a411bcf2c3b268292323327531210d70db72" Nov 02 23:10:16.462196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.458926 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:16.462196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.458998 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:16.487433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.487364 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:16.606074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.606023 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\" (UniqueName: \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\") pod \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " Nov 02 23:10:16.606353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.606127 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\") pod \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " Nov 02 23:10:16.606353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.606195 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\") pod \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\" (UID: \"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be\") " Nov 02 23:10:16.608496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.608463 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca" pod "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" (UID: "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:16.621205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.613351 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca" pod "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" (UID: "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:16.624680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.624659 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45" pod "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" (UID: "9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be"). InnerVolumeSpecName "kube-api-access-k5s45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:16.707531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.707498 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5s45\" (UniqueName: \"kubernetes.io/projected/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-kube-api-access-k5s45\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:16.707531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.707524 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:16.707531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:16.707535 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.054920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.054884 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv"] Nov 02 23:10:17.061501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.061476 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdda8137-z2qtv"] Nov 02 23:10:17.239576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.239535 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" Nov 02 23:10:17.240066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.239596 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds" event={"ID":"9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be","Type":"ContainerDied","Data":"37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1"} Nov 02 23:10:17.240066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.239684 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37129c8df402a7dff75e2409ce8389ffb357351233d70f5530625244793147e1" Nov 02 23:10:17.323492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.323400 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:10:17.323492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.323469 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:10:17.359283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.358924 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:17.360225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.360042 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:17.361208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.361130 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:10:17.443930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.443897 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:17.457921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.457892 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 23:10:17.458068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.457950 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 23:10:17.560061 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:10:17.560025 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:10:17.634249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.633539 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\") pod \"307c5897-ae78-484c-8ddc-801d53a138fe\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " Nov 02 23:10:17.634249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.633810 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\") pod \"307c5897-ae78-484c-8ddc-801d53a138fe\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " Nov 02 23:10:17.634249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.634006 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\" (UniqueName: \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\") pod \"307c5897-ae78-484c-8ddc-801d53a138fe\" (UID: \"307c5897-ae78-484c-8ddc-801d53a138fe\") " Nov 02 23:10:17.637648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.637600 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca" pod "307c5897-ae78-484c-8ddc-801d53a138fe" (UID: "307c5897-ae78-484c-8ddc-801d53a138fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:17.637960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.637926 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh" pod "307c5897-ae78-484c-8ddc-801d53a138fe" (UID: "307c5897-ae78-484c-8ddc-801d53a138fe"). InnerVolumeSpecName "kube-api-access-dq5dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:17.647312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.647280 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca" pod "307c5897-ae78-484c-8ddc-801d53a138fe" (UID: "307c5897-ae78-484c-8ddc-801d53a138fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:17.698296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.698258 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0936acdd-3670-43c3-8266-36446b6cd143" path="/var/lib/kubelet/pods/0936acdd-3670-43c3-8266-36446b6cd143/volumes" Nov 02 23:10:17.736389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.736349 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5dh\" (UniqueName: \"kubernetes.io/projected/307c5897-ae78-484c-8ddc-801d53a138fe-kube-api-access-dq5dh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.736389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.736391 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.737199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.736407 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/307c5897-ae78-484c-8ddc-801d53a138fe-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:17.975433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:17.975395 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:18.248382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.248296 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" Nov 02 23:10:18.248382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.248340 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b" event={"ID":"307c5897-ae78-484c-8ddc-801d53a138fe","Type":"ContainerDied","Data":"4b82cd691bd079787cdd5c484832de35aa23a1abbd279a48f3c9ba626a969dd9"} Nov 02 23:10:18.248991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.248395 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b82cd691bd079787cdd5c484832de35aa23a1abbd279a48f3c9ba626a969dd9" Nov 02 23:10:18.462426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.462395 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:18.462834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.462804 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:18.463693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:18.463667 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:10:22.949334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:22.949300 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:23.079159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:23.073903 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr"] Nov 02 23:10:23.105277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:23.105246 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6009e1a2-q8fzr"] Nov 02 23:10:23.697883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:23.697840 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cbf5f5-9071-4a3f-ac46-e4d81c284274" path="/var/lib/kubelet/pods/78cbf5f5-9071-4a3f-ac46-e4d81c284274/volumes" Nov 02 23:10:24.714782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.714753 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6"] Nov 02 23:10:24.715381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715365 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c7292e94-19f0-4336-a312-39ee18b66d5a" containerName="revoke-keys" Nov 02 23:10:24.715425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715382 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7292e94-19f0-4336-a312-39ee18b66d5a" containerName="revoke-keys" Nov 02 23:10:24.715425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715406 4494 cpu_manager.go:414] "containerMap: removing container" podUID="307c5897-ae78-484c-8ddc-801d53a138fe" containerName="revoke-keys" Nov 02 23:10:24.715425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715410 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307c5897-ae78-484c-8ddc-801d53a138fe" containerName="revoke-keys" Nov 02 23:10:24.715528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715431 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" containerName="revoke-keys" Nov 02 23:10:24.715528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715436 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" containerName="revoke-keys" Nov 02 23:10:24.715591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715535 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7292e94-19f0-4336-a312-39ee18b66d5a" containerName="revoke-keys" Nov 02 23:10:24.715591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715547 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="307c5897-ae78-484c-8ddc-801d53a138fe" containerName="revoke-keys" Nov 02 23:10:24.715591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.715556 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" containerName="revoke-keys" Nov 02 23:10:24.719254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.719235 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.776590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.776563 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6"] Nov 02 23:10:24.821811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.821785 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:10:24.821982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.821840 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:10:24.838571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.838538 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.838718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.838577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpps\" (UniqueName: \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.838718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.838683 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.941317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.941283 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.941509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.941328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpps\" (UniqueName: \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.941509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.941391 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.942256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.942232 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:24.944031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:24.944007 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:25.003042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:25.002973 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpps\" (UniqueName: \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\") pod \"maas-api-revoke-sub-434942fd-5b8l6\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:25.032045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:25.032013 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:25.411226 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:25.411132 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bab6e0_cac2_43d1_bae4_aa178d8239bf.slice/crio-0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b WatchSource:0}: Error finding container 0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b: Status 404 returned error can't find the container with id 0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b Nov 02 23:10:25.411342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:25.411233 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6"] Nov 02 23:10:26.310285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:26.310243 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" event={"ID":"89bab6e0-cac2-43d1-bae4-aa178d8239bf","Type":"ContainerStarted","Data":"4640b9f2ef075b45e465e2dfba2c4a267b7024288730c381992002663955cb3c"} Nov 02 23:10:26.310741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:26.310313 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" event={"ID":"89bab6e0-cac2-43d1-bae4-aa178d8239bf","Type":"ContainerStarted","Data":"0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b"} Nov 02 23:10:26.431610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:26.431553 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" podStartSLOduration=2.43153893 podStartE2EDuration="2.43153893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:26.430082012 +0000 UTC m=+1340.859951872" watchObservedRunningTime="2026-11-02 23:10:26.43153893 +0000 UTC m=+1340.861408797" Nov 02 23:10:27.319389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:27.319351 4494 generic.go:356] "Generic (PLEG): container finished" podID="89bab6e0-cac2-43d1-bae4-aa178d8239bf" containerID="4640b9f2ef075b45e465e2dfba2c4a267b7024288730c381992002663955cb3c" exitCode=0 Nov 02 23:10:27.319803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:27.319398 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" event={"ID":"89bab6e0-cac2-43d1-bae4-aa178d8239bf","Type":"ContainerDied","Data":"4640b9f2ef075b45e465e2dfba2c4a267b7024288730c381992002663955cb3c"} Nov 02 23:10:27.852523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:27.852488 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:28.468350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.468326 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:28.608728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.608641 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\") pod \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " Nov 02 23:10:28.608893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.608771 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\" (UniqueName: \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\") pod \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " Nov 02 23:10:28.608893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.608856 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\") pod \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\" (UID: \"89bab6e0-cac2-43d1-bae4-aa178d8239bf\") " Nov 02 23:10:28.609436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.609407 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca" pod "89bab6e0-cac2-43d1-bae4-aa178d8239bf" (UID: "89bab6e0-cac2-43d1-bae4-aa178d8239bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:28.609601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.609546 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:28.611549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.611523 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps" pod "89bab6e0-cac2-43d1-bae4-aa178d8239bf" (UID: "89bab6e0-cac2-43d1-bae4-aa178d8239bf"). InnerVolumeSpecName "kube-api-access-clpps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:28.613274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.613255 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca" pod "89bab6e0-cac2-43d1-bae4-aa178d8239bf" (UID: "89bab6e0-cac2-43d1-bae4-aa178d8239bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:28.711050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.711019 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpps\" (UniqueName: \"kubernetes.io/projected/89bab6e0-cac2-43d1-bae4-aa178d8239bf-kube-api-access-clpps\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:28.711250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:28.711056 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bab6e0-cac2-43d1-bae4-aa178d8239bf-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:29.335533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:29.335499 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" event={"ID":"89bab6e0-cac2-43d1-bae4-aa178d8239bf","Type":"ContainerDied","Data":"0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b"} Nov 02 23:10:29.335712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:29.335537 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6" Nov 02 23:10:29.335712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:29.335545 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f992e3121403f45abb536855abf736656e20b9bb2fe64b2976d650efa35023b" Nov 02 23:10:30.382893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:30.382862 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:10:31.253962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.250873 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq"] Nov 02 23:10:31.253962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.251865 4494 cpu_manager.go:414] "containerMap: removing container" podUID="89bab6e0-cac2-43d1-bae4-aa178d8239bf" containerName="revoke-keys" Nov 02 23:10:31.253962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.251884 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bab6e0-cac2-43d1-bae4-aa178d8239bf" containerName="revoke-keys" Nov 02 23:10:31.253962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.252015 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bab6e0-cac2-43d1-bae4-aa178d8239bf" containerName="revoke-keys" Nov 02 23:10:31.318194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.314429 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq"] Nov 02 23:10:31.318194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.314562 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.425198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.421725 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.425198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.421775 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpwc\" (UniqueName: \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.425198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.421829 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.526245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.505283 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:10:31.526245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.505473 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:10:31.578999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.578126 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.578999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.578190 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpwc\" (UniqueName: \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.578999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.578245 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.601196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.582682 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.622636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.622559 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.897246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.896868 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpwc\" (UniqueName: \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\") pod \"maas-api-revoke-sub-959a96de-dtnsq\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:31.933198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:31.932527 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:32.633406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:32.633370 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx"] Nov 02 23:10:32.836222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:32.821529 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:32.862207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:32.855057 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx"] Nov 02 23:10:32.862207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:32.855143 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq"] Nov 02 23:10:33.066346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.066256 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.077546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.077503 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwrc\" (UniqueName: \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.166197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.157392 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.271053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.271009 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.281479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.281447 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.305695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.282604 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.305695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.289609 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.306084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.306057 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwrc\" (UniqueName: \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.402721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.402689 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwrc\" (UniqueName: \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\") pod \"maas-api-revoke-sub-02d1c509-b5bsx\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.659530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.659504 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:33.820833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:33.820790 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" event={"ID":"1c63d5fc-6676-4890-8e71-46c2db2028b5","Type":"ContainerStarted","Data":"e2f0f29b424924c2d1d19e863524a4c7e8992698919663b9806b8dba27332bde"} Nov 02 23:10:34.074517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.073096 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824"] Nov 02 23:10:34.292251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.286341 4494 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 02 23:10:34.292251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.288918 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ec36fc4-2t824"] Nov 02 23:10:34.490631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.479432 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx"] Nov 02 23:10:34.833343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.833302 4494 generic.go:356] "Generic (PLEG): container finished" podID="1c63d5fc-6676-4890-8e71-46c2db2028b5" containerID="90ba2bae7585e6e43540e5e0762e9f203e7964aa23cda6a0201ce77e5c81a1dc" exitCode=0 Nov 02 23:10:34.833797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.833348 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" event={"ID":"1c63d5fc-6676-4890-8e71-46c2db2028b5","Type":"ContainerDied","Data":"90ba2bae7585e6e43540e5e0762e9f203e7964aa23cda6a0201ce77e5c81a1dc"} Nov 02 23:10:34.835184 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:34.835129 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" event={"ID":"3e1e57d5-4bc6-4bd0-87e5-a3b866099806","Type":"ContainerStarted","Data":"2b3124010e457244777ff850cbaaf53737c0277114fc93d27adff9f0537b6802"} Nov 02 23:10:35.325210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.325099 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:35.476364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.476328 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw"] Nov 02 23:10:35.489964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.489935 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.524810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.524772 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw"] Nov 02 23:10:35.616445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.616405 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.616625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.616606 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbcts\" (UniqueName: \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.616673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.616644 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.695637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.695606 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac28ec94-5b4c-4565-b878-414be67b6396" path="/var/lib/kubelet/pods/ac28ec94-5b4c-4565-b878-414be67b6396/volumes" Nov 02 23:10:35.718419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.718375 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.718576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.718498 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbcts\" (UniqueName: \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.718576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.718526 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.719006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.718986 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.720928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.720907 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.780790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.780757 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbcts\" (UniqueName: \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\") pod \"maas-api-revoke-sub-1dafbca2-w7dgw\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.806390 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.806368 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:35.845144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.845062 4494 generic.go:356] "Generic (PLEG): container finished" podID="3e1e57d5-4bc6-4bd0-87e5-a3b866099806" containerID="6a7ba5765d4baa9533ba1f4d6549fe744c2ea4996cbdb5beacfa6cddffe652cf" exitCode=0 Nov 02 23:10:35.845748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:35.845724 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" event={"ID":"3e1e57d5-4bc6-4bd0-87e5-a3b866099806","Type":"ContainerDied","Data":"6a7ba5765d4baa9533ba1f4d6549fe744c2ea4996cbdb5beacfa6cddffe652cf"} Nov 02 23:10:36.014339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.014313 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:36.123481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.123392 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\") pod \"1c63d5fc-6676-4890-8e71-46c2db2028b5\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " Nov 02 23:10:36.123481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.123435 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\") pod \"1c63d5fc-6676-4890-8e71-46c2db2028b5\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " Nov 02 23:10:36.123688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.123602 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\" (UniqueName: \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\") pod \"1c63d5fc-6676-4890-8e71-46c2db2028b5\" (UID: \"1c63d5fc-6676-4890-8e71-46c2db2028b5\") " Nov 02 23:10:36.123830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.123806 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca" pod "1c63d5fc-6676-4890-8e71-46c2db2028b5" (UID: "1c63d5fc-6676-4890-8e71-46c2db2028b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:36.124106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.124078 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:36.125701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.125677 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc" pod "1c63d5fc-6676-4890-8e71-46c2db2028b5" (UID: "1c63d5fc-6676-4890-8e71-46c2db2028b5"). InnerVolumeSpecName "kube-api-access-ktpwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:36.125845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.125827 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca" pod "1c63d5fc-6676-4890-8e71-46c2db2028b5" (UID: "1c63d5fc-6676-4890-8e71-46c2db2028b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:36.187639 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:36.187602 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302b3db7_b2d9_4faa_878d_d023b6706e44.slice/crio-96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4 WatchSource:0}: Error finding container 96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4: Status 404 returned error can't find the container with id 96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4 Nov 02 23:10:36.188319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.188294 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw"] Nov 02 23:10:36.225604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.225580 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c63d5fc-6676-4890-8e71-46c2db2028b5-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:36.225604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.225605 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktpwc\" (UniqueName: \"kubernetes.io/projected/1c63d5fc-6676-4890-8e71-46c2db2028b5-kube-api-access-ktpwc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:36.854076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.853972 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" event={"ID":"302b3db7-b2d9-4faa-878d-d023b6706e44","Type":"ContainerStarted","Data":"f4ec5fedaa84449423b0c74683198c2f06bf05e0521ce313cc9a5b1f283373ac"} Nov 02 23:10:36.854076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.854051 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" event={"ID":"302b3db7-b2d9-4faa-878d-d023b6706e44","Type":"ContainerStarted","Data":"96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4"} Nov 02 23:10:36.855383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.855365 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" Nov 02 23:10:36.855448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.855407 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq" event={"ID":"1c63d5fc-6676-4890-8e71-46c2db2028b5","Type":"ContainerDied","Data":"e2f0f29b424924c2d1d19e863524a4c7e8992698919663b9806b8dba27332bde"} Nov 02 23:10:36.855497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.855448 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2f0f29b424924c2d1d19e863524a4c7e8992698919663b9806b8dba27332bde" Nov 02 23:10:36.990165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:36.983882 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" podStartSLOduration=1.983862349 podStartE2EDuration="1.983862349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:36.977945401 +0000 UTC m=+1351.407815263" watchObservedRunningTime="2026-11-02 23:10:36.983862349 +0000 UTC m=+1351.413732210" Nov 02 23:10:37.034489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.034467 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:37.136426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.136374 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\") pod \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " Nov 02 23:10:37.136546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.136530 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\") pod \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " Nov 02 23:10:37.136648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.136629 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\" (UniqueName: \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\") pod \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\" (UID: \"3e1e57d5-4bc6-4bd0-87e5-a3b866099806\") " Nov 02 23:10:37.136776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.136751 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca" pod "3e1e57d5-4bc6-4bd0-87e5-a3b866099806" (UID: "3e1e57d5-4bc6-4bd0-87e5-a3b866099806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:37.137027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.137007 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:37.138749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.138727 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca" pod "3e1e57d5-4bc6-4bd0-87e5-a3b866099806" (UID: "3e1e57d5-4bc6-4bd0-87e5-a3b866099806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:37.138820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.138727 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc" pod "3e1e57d5-4bc6-4bd0-87e5-a3b866099806" (UID: "3e1e57d5-4bc6-4bd0-87e5-a3b866099806"). InnerVolumeSpecName "kube-api-access-xhwrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:37.238649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.238606 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:37.238649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.238643 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwrc\" (UniqueName: \"kubernetes.io/projected/3e1e57d5-4bc6-4bd0-87e5-a3b866099806-kube-api-access-xhwrc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:37.863931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.863901 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" Nov 02 23:10:37.864411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.863937 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx" event={"ID":"3e1e57d5-4bc6-4bd0-87e5-a3b866099806","Type":"ContainerDied","Data":"2b3124010e457244777ff850cbaaf53737c0277114fc93d27adff9f0537b6802"} Nov 02 23:10:37.864411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.863996 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3124010e457244777ff850cbaaf53737c0277114fc93d27adff9f0537b6802" Nov 02 23:10:37.865574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.865547 4494 generic.go:356] "Generic (PLEG): container finished" podID="302b3db7-b2d9-4faa-878d-d023b6706e44" containerID="f4ec5fedaa84449423b0c74683198c2f06bf05e0521ce313cc9a5b1f283373ac" exitCode=0 Nov 02 23:10:37.865679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:37.865608 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" event={"ID":"302b3db7-b2d9-4faa-878d-d023b6706e44","Type":"ContainerDied","Data":"f4ec5fedaa84449423b0c74683198c2f06bf05e0521ce313cc9a5b1f283373ac"} Nov 02 23:10:39.029371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.029345 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:39.162326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.162302 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\") pod \"302b3db7-b2d9-4faa-878d-d023b6706e44\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " Nov 02 23:10:39.162497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.162373 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\") pod \"302b3db7-b2d9-4faa-878d-d023b6706e44\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " Nov 02 23:10:39.162593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.162576 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\" (UniqueName: \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\") pod \"302b3db7-b2d9-4faa-878d-d023b6706e44\" (UID: \"302b3db7-b2d9-4faa-878d-d023b6706e44\") " Nov 02 23:10:39.162810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.162786 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca" pod "302b3db7-b2d9-4faa-878d-d023b6706e44" (UID: "302b3db7-b2d9-4faa-878d-d023b6706e44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:39.163064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.163046 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:39.164981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.164957 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts" pod "302b3db7-b2d9-4faa-878d-d023b6706e44" (UID: "302b3db7-b2d9-4faa-878d-d023b6706e44"). InnerVolumeSpecName "kube-api-access-jbcts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:39.165139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.165115 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca" pod "302b3db7-b2d9-4faa-878d-d023b6706e44" (UID: "302b3db7-b2d9-4faa-878d-d023b6706e44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:39.264710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.264674 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbcts\" (UniqueName: \"kubernetes.io/projected/302b3db7-b2d9-4faa-878d-d023b6706e44-kube-api-access-jbcts\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:39.264710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.264707 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302b3db7-b2d9-4faa-878d-d023b6706e44-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:39.881689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.881658 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" Nov 02 23:10:39.881837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.881695 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw" event={"ID":"302b3db7-b2d9-4faa-878d-d023b6706e44","Type":"ContainerDied","Data":"96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4"} Nov 02 23:10:39.881837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:39.881742 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96e3d692fbd20c1e5e62186b1528071a24ab2121dea72296d93abb87f3f065a4" Nov 02 23:10:40.253880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:40.253840 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:41.955434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.955402 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w"] Nov 02 23:10:41.956104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956088 4494 cpu_manager.go:414] "containerMap: removing container" podUID="302b3db7-b2d9-4faa-878d-d023b6706e44" containerName="revoke-keys" Nov 02 23:10:41.956166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956106 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302b3db7-b2d9-4faa-878d-d023b6706e44" containerName="revoke-keys" Nov 02 23:10:41.956166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956120 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1c63d5fc-6676-4890-8e71-46c2db2028b5" containerName="revoke-keys" Nov 02 23:10:41.956166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956125 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c63d5fc-6676-4890-8e71-46c2db2028b5" containerName="revoke-keys" Nov 02 23:10:41.956166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956150 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3e1e57d5-4bc6-4bd0-87e5-a3b866099806" containerName="revoke-keys" Nov 02 23:10:41.956166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956155 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1e57d5-4bc6-4bd0-87e5-a3b866099806" containerName="revoke-keys" Nov 02 23:10:41.956353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956267 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c63d5fc-6676-4890-8e71-46c2db2028b5" containerName="revoke-keys" Nov 02 23:10:41.956353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956281 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1e57d5-4bc6-4bd0-87e5-a3b866099806" containerName="revoke-keys" Nov 02 23:10:41.956353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.956292 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="302b3db7-b2d9-4faa-878d-d023b6706e44" containerName="revoke-keys" Nov 02 23:10:41.959844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:41.959828 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.016379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.016349 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w"] Nov 02 23:10:42.062132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.062091 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:10:42.062343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.062142 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:10:42.103318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.103282 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.103547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.103446 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.103623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.103583 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbp54\" (UniqueName: \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.204819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.204785 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.204989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.204891 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.204989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.204954 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbp54\" (UniqueName: \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.205609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.205554 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.207367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.207346 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.270092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.270057 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbp54\" (UniqueName: \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\") pod \"maas-api-revoke-sub-71fab6c9-8sm9w\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.273914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.273893 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:42.655634 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:42.655600 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb5afbe_790c_476d_91e1_d52c9cbf01f8.slice/crio-0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07 WatchSource:0}: Error finding container 0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07: Status 404 returned error can't find the container with id 0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07 Nov 02 23:10:42.655801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.655734 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w"] Nov 02 23:10:42.905423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:42.905348 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" event={"ID":"5cb5afbe-790c-476d-91e1-d52c9cbf01f8","Type":"ContainerStarted","Data":"0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07"} Nov 02 23:10:43.365122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:43.365076 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:43.511739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:43.511696 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:43.914775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:43.914740 4494 generic.go:356] "Generic (PLEG): container finished" podID="5cb5afbe-790c-476d-91e1-d52c9cbf01f8" containerID="7de8fae823b8725ce1a0736e96aa590ace505fd6b63dc7ea856b59a834fb9f72" exitCode=0 Nov 02 23:10:43.914933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:43.914780 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" event={"ID":"5cb5afbe-790c-476d-91e1-d52c9cbf01f8","Type":"ContainerDied","Data":"7de8fae823b8725ce1a0736e96aa590ace505fd6b63dc7ea856b59a834fb9f72"} Nov 02 23:10:45.080227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.080196 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:45.246680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.246603 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\") pod \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " Nov 02 23:10:45.246680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.246658 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\" (UniqueName: \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\") pod \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " Nov 02 23:10:45.246921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.246719 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\") pod \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\" (UID: \"5cb5afbe-790c-476d-91e1-d52c9cbf01f8\") " Nov 02 23:10:45.246921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.246751 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:45.247306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.247274 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca" pod "5cb5afbe-790c-476d-91e1-d52c9cbf01f8" (UID: "5cb5afbe-790c-476d-91e1-d52c9cbf01f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:10:45.249234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.249193 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54" pod "5cb5afbe-790c-476d-91e1-d52c9cbf01f8" (UID: "5cb5afbe-790c-476d-91e1-d52c9cbf01f8"). InnerVolumeSpecName "kube-api-access-sbp54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:10:45.249539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.249510 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca" pod "5cb5afbe-790c-476d-91e1-d52c9cbf01f8" (UID: "5cb5afbe-790c-476d-91e1-d52c9cbf01f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:10:45.349112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.349057 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:45.349112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.349106 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbp54\" (UniqueName: \"kubernetes.io/projected/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-kube-api-access-sbp54\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:45.349420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.349126 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb5afbe-790c-476d-91e1-d52c9cbf01f8-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:10:45.933585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.933548 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" event={"ID":"5cb5afbe-790c-476d-91e1-d52c9cbf01f8","Type":"ContainerDied","Data":"0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07"} Nov 02 23:10:45.933585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.933572 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w" Nov 02 23:10:45.933792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:45.933602 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0923250a47e0a7e67a248c0780a1f82a29f53c210aba9e806ff52a8933fa3c07" Nov 02 23:10:47.562891 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:10:47.562851 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:10:48.794865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:48.794796 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:49.558683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.558650 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:10:49.559277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.559261 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5cb5afbe-790c-476d-91e1-d52c9cbf01f8" containerName="revoke-keys" Nov 02 23:10:49.559277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.559277 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb5afbe-790c-476d-91e1-d52c9cbf01f8" containerName="revoke-keys" Nov 02 23:10:49.559588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.559565 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb5afbe-790c-476d-91e1-d52c9cbf01f8" containerName="revoke-keys" Nov 02 23:10:49.563322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.563305 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.644453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.644415 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a7a67875-openshift-default-dockercfg-55dcr\"" Nov 02 23:10:49.648902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.648865 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:49.674955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.674918 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:10:49.761281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.757288 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.761629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.761596 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.761791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.761773 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.762029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.762010 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.765944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.762328 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.766165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.766137 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgh9\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.766389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.766368 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.766506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.766493 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.766624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.766612 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.871276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.871147 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.871917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.871872 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgh9\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.876204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.872091 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.876456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.876430 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.876614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.876598 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.881203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.881147 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.881386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.872466 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.881495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.871672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:49.903742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.903700 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.904968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.904108 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.905266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.905241 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.905453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.905427 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.905670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.905649 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.906240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.904897 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.907097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.906701 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.925742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.918295 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.926713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.926564 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.950788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.950757 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgh9\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:49.968783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:49.968753 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\") pod \"e2e-mt-a7a67875-openshift-default-7465746989-fp4c6\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:50.177290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.177251 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:50.360924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.360894 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:50.507760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.507564 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:50.704196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.695411 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:10:50.736040 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:50.735933 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ff9079_a428_4714_ad29_75ed3beb3768.slice/crio-421f490306a18571ccb0d9b8b65ba61631bb229ae57d9bea4602b440fb16a68a WatchSource:0}: Error finding container 421f490306a18571ccb0d9b8b65ba61631bb229ae57d9bea4602b440fb16a68a: Status 404 returned error can't find the container with id 421f490306a18571ccb0d9b8b65ba61631bb229ae57d9bea4602b440fb16a68a Nov 02 23:10:50.769546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.769503 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:50.769790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.769766 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:50.769923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.769906 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:50.974936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.974904 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerStarted","Data":"d30bb4641382cbfbc16be521b4dc36a157a2a612c22ddb4a4b78eb987a2d8455"} Nov 02 23:10:50.975430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:50.974959 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerStarted","Data":"421f490306a18571ccb0d9b8b65ba61631bb229ae57d9bea4602b440fb16a68a"} Nov 02 23:10:51.096729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.096626 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podStartSLOduration=2.096611334 podStartE2EDuration="2.096611334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:51.096234877 +0000 UTC m=+1365.526104739" watchObservedRunningTime="2026-11-02 23:10:51.096611334 +0000 UTC m=+1365.526481195" Nov 02 23:10:51.178161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.178125 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:51.179585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.179564 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:10:51.179660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.179614 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:10:51.606908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.606874 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:10:51.611316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.611297 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652028 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652085 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652246 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652324 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652500 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652542 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtst\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652619 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652663 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.652714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.652696 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.782932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.782890 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.782976 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783020 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783068 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783145 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtst\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783303 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783360 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.783409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.783389 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.784744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.784709 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:10:51.794808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.788018 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.794808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.788377 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.794808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.789159 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.794808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.789454 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.794808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.789701 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.797749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.797710 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.798297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.798272 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.850468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.850431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.850641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.850605 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtst\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:51.926978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:51.926943 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:52.182166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.182095 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:52.377635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.377605 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:10:52.378920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.378880 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.379018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.378969 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.379081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.379013 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:10:52.993297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.993250 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" event={"ID":"f5987d37-eda3-4816-a39e-3bd4229da31a","Type":"ContainerStarted","Data":"6ccf2baebfae2408b577abde90e4be39a6e5fe9cfa0f5ab39ef7b1dbbafba5d0"} Nov 02 23:10:52.993513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.993315 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" event={"ID":"f5987d37-eda3-4816-a39e-3bd4229da31a","Type":"ContainerStarted","Data":"84c0809d0efe57eb372192ad1a9879a3a96bb35096d6c32529d8ff1d4805e28f"} Nov 02 23:10:52.993841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.993817 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:52.995340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:52.995317 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:10:53.126555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:53.126500 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podStartSLOduration=2.126486721 podStartE2EDuration="2.126486721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:53.124188774 +0000 UTC m=+1367.554058627" watchObservedRunningTime="2026-11-02 23:10:53.126486721 +0000 UTC m=+1367.556356588" Nov 02 23:10:53.941541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:53.941505 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:10:53.942370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:53.942345 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:53.942588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:53.942561 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:54.072870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.072773 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:54.237132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.237010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:54.391747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.391693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:54.549912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.549722 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:54.928940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.928905 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:54.929275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:54.929243 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:55.337263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:55.327676 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:55.493204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:55.493020 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:55.682204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:55.680717 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:55.930203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:55.929335 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:55.930203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:55.929399 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:56.954320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:56.941932 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:56.954320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:56.941990 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:57.498522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:57.498482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:57.927662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:57.927618 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:57.927855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:57.927683 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:58.156827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:58.156777 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:58.375258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:58.341857 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:10:58.855768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:58.854463 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:10:58.998852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:58.998802 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:58.999328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:58.999293 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:10:59.012289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.009568 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.018848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.018814 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:10:59.113011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.112921 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a7a67875-dockercfg-44n9v\"" Nov 02 23:10:59.113011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.112923 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a7a67875\"" Nov 02 23:10:59.168107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.168067 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.168579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.168147 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmhz\" (UniqueName: \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.269732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.269695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.269919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.269786 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmhz\" (UniqueName: \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.270474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.270455 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.356065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.356034 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmhz\" (UniqueName: \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.364899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.364831 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:10:59.803353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.803323 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:10:59.803488 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:10:59.803459 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d846c8_c350_4748_bde1_9d8e593c58d5.slice/crio-0a795c38d7818fd8495b6cd5b163a4c04950d8969b3e492d1ce0e48671c3b195 WatchSource:0}: Error finding container 0a795c38d7818fd8495b6cd5b163a4c04950d8969b3e492d1ce0e48671c3b195: Status 404 returned error can't find the container with id 0a795c38d7818fd8495b6cd5b163a4c04950d8969b3e492d1ce0e48671c3b195 Nov 02 23:10:59.927276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.927245 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:10:59.927415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:10:59.927296 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:11:00.421482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.421445 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" event={"ID":"72d846c8-c350-4748-bde1-9d8e593c58d5","Type":"ContainerStarted","Data":"1fb56813ae3836294fc6d66ddb89d79fc60cf43072f24a9a4946d05bcb83a88f"} Nov 02 23:11:00.421908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.421513 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" event={"ID":"72d846c8-c350-4748-bde1-9d8e593c58d5","Type":"ContainerStarted","Data":"0a795c38d7818fd8495b6cd5b163a4c04950d8969b3e492d1ce0e48671c3b195"} Nov 02 23:11:00.421908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.421543 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:11:00.553681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.553545 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" podStartSLOduration=2.553519301 podStartE2EDuration="2.553519301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:00.537982811 +0000 UTC m=+1374.967852681" watchObservedRunningTime="2026-11-02 23:11:00.553519301 +0000 UTC m=+1374.983389163" Nov 02 23:11:00.936789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.933466 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 23:11:00.936789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:00.933575 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 23:11:02.084952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.084920 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:02.157063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.157031 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:11:02.157474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.157450 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:11:02.158519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.158500 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:11:02.237650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.237614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:02.385322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.385241 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:02.755752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:02.755720 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:03.033222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:03.033115 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:06.155802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:06.155770 4494 scope.go:122] "RemoveContainer" containerID="4be9f8732b2b449b1179cfe656b0910b971aa9a87f3f1fc2491c9702d5ccd9c5" Nov 02 23:11:06.165724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:06.165704 4494 scope.go:122] "RemoveContainer" containerID="aa733f0de8d8231b3a67983de3ec73dc79e3062effc106914943a5b0a2e93e8c" Nov 02 23:11:06.174851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:06.174830 4494 scope.go:122] "RemoveContainer" containerID="1e12e451885e49d4a36ec272a5cb2bb9f614279fbce96ded1e59f79d50ff78e2" Nov 02 23:11:06.183921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:06.183901 4494 scope.go:122] "RemoveContainer" containerID="79ea11144c219de04130d57b37a7b62504e1f12b08734e534402780af69a633c" Nov 02 23:11:07.764932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:07.764904 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:09.099388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.099340 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw"] Nov 02 23:11:09.105380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.105361 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.160400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.160369 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw"] Nov 02 23:11:09.205283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.205248 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:09.205505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.205456 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:11:09.205902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.205880 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8m6\" (UniqueName: \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.205954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.205936 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.206146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.206125 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.307732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.307695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.307927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.307795 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8m6\" (UniqueName: \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.307927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.307835 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.308546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.308514 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.310600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.310566 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.369978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.369895 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8m6\" (UniqueName: \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\") pod \"maas-api-revoke-sub-ef02a122-tzppw\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.419485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.419448 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:09.800552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:09.800522 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw"] Nov 02 23:11:09.800748 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:11:09.800723 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50650c51_3047_4ad6_85f9_631a06dbc683.slice/crio-d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b WatchSource:0}: Error finding container d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b: Status 404 returned error can't find the container with id d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b Nov 02 23:11:10.513320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:10.513286 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" event={"ID":"50650c51-3047-4ad6-85f9-631a06dbc683","Type":"ContainerStarted","Data":"77e772384a9df10e7b2c63a93126aa2a7d50894b24e5a1f659d229ddbf568781"} Nov 02 23:11:10.513741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:10.513340 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" event={"ID":"50650c51-3047-4ad6-85f9-631a06dbc683","Type":"ContainerStarted","Data":"d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b"} Nov 02 23:11:10.633756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:10.633701 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" podStartSLOduration=1.633685082 podStartE2EDuration="1.633685082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:10.630457306 +0000 UTC m=+1385.060327170" watchObservedRunningTime="2026-11-02 23:11:10.633685082 +0000 UTC m=+1385.063554950" Nov 02 23:11:11.434698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:11.434664 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:11:11.521775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:11.521736 4494 generic.go:356] "Generic (PLEG): container finished" podID="50650c51-3047-4ad6-85f9-631a06dbc683" containerID="77e772384a9df10e7b2c63a93126aa2a7d50894b24e5a1f659d229ddbf568781" exitCode=0 Nov 02 23:11:11.522209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:11.521815 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" event={"ID":"50650c51-3047-4ad6-85f9-631a06dbc683","Type":"ContainerDied","Data":"77e772384a9df10e7b2c63a93126aa2a7d50894b24e5a1f659d229ddbf568781"} Nov 02 23:11:12.206822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:12.206786 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:11:13.004819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.004777 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:13.069447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.069407 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:13.212730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.212690 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\") pod \"50650c51-3047-4ad6-85f9-631a06dbc683\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " Nov 02 23:11:13.213118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.213094 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\" (UniqueName: \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\") pod \"50650c51-3047-4ad6-85f9-631a06dbc683\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " Nov 02 23:11:13.213402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.213382 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\") pod \"50650c51-3047-4ad6-85f9-631a06dbc683\" (UID: \"50650c51-3047-4ad6-85f9-631a06dbc683\") " Nov 02 23:11:13.215228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.215203 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca" pod "50650c51-3047-4ad6-85f9-631a06dbc683" (UID: "50650c51-3047-4ad6-85f9-631a06dbc683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:13.235357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.234999 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6" pod "50650c51-3047-4ad6-85f9-631a06dbc683" (UID: "50650c51-3047-4ad6-85f9-631a06dbc683"). InnerVolumeSpecName "kube-api-access-sf8m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:13.237826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.237782 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca" pod "50650c51-3047-4ad6-85f9-631a06dbc683" (UID: "50650c51-3047-4ad6-85f9-631a06dbc683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:13.315481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.315441 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:13.315481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.315483 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50650c51-3047-4ad6-85f9-631a06dbc683-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:13.315481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.315499 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8m6\" (UniqueName: \"kubernetes.io/projected/50650c51-3047-4ad6-85f9-631a06dbc683-kube-api-access-sf8m6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:13.579274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.579163 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" event={"ID":"50650c51-3047-4ad6-85f9-631a06dbc683","Type":"ContainerDied","Data":"d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b"} Nov 02 23:11:13.579532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.579511 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d50596b197bf190e02ad02755f7a7489a36534fb9d857930c1f98c545ff2be2b" Nov 02 23:11:13.579709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:13.579695 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw" Nov 02 23:11:15.078197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:15.076689 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq"] Nov 02 23:11:15.130200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:15.128931 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b860b205-dvdpq"] Nov 02 23:11:15.852198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:15.846965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:15.933492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:15.933451 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbdafbe-3408-4de5-a9fe-27bd47cdcacd" path="/var/lib/kubelet/pods/abbdafbe-3408-4de5-a9fe-27bd47cdcacd/volumes" Nov 02 23:11:17.672256 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:11:17.671485 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:11:18.478566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:18.478535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:21.602884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:21.602843 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:21.877547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:21.877459 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:23.463201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:23.463141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:25.833789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:25.833759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:28.245472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:28.245441 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:28.406992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:28.406951 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:28.556468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:28.556224 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:33.451500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:33.451462 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:38.360377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:38.360338 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:43.428728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:43.428695 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:47.626553 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:11:47.626514 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:11:48.991374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:48.991332 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:49.568396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.568360 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk"] Nov 02 23:11:49.570115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.570087 4494 cpu_manager.go:414] "containerMap: removing container" podUID="50650c51-3047-4ad6-85f9-631a06dbc683" containerName="revoke-keys" Nov 02 23:11:49.573418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.573389 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50650c51-3047-4ad6-85f9-631a06dbc683" containerName="revoke-keys" Nov 02 23:11:49.573906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.573754 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="50650c51-3047-4ad6-85f9-631a06dbc683" containerName="revoke-keys" Nov 02 23:11:49.578255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.578229 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.604969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.604944 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk"] Nov 02 23:11:49.656037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.654380 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:11:49.656037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.655943 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:49.677563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.677533 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.677721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.677625 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws7g7\" (UniqueName: \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.677841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.677825 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.778991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.778952 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.779267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.779056 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.779267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.779086 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws7g7\" (UniqueName: \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.779954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.779930 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.781722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.781702 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.855993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.855910 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws7g7\" (UniqueName: \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\") pod \"maas-api-revoke-sub-8c4c3385-c7slk\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:49.891983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:49.891947 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:50.294282 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:11:50.294236 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edcd3d3_661d_4423_a151_75b803db33bf.slice/crio-47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da WatchSource:0}: Error finding container 47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da: Status 404 returned error can't find the container with id 47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da Nov 02 23:11:50.294682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:50.294656 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk"] Nov 02 23:11:51.240148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:51.240100 4494 generic.go:356] "Generic (PLEG): container finished" podID="9edcd3d3-661d-4423-a151-75b803db33bf" containerID="ac99c88afbaf0ddb36c61525702da02f74087a6dae0cc0826808ccb01d63a47a" exitCode=0 Nov 02 23:11:51.240421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:51.240217 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" event={"ID":"9edcd3d3-661d-4423-a151-75b803db33bf","Type":"ContainerDied","Data":"ac99c88afbaf0ddb36c61525702da02f74087a6dae0cc0826808ccb01d63a47a"} Nov 02 23:11:51.240421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:51.240279 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" event={"ID":"9edcd3d3-661d-4423-a151-75b803db33bf","Type":"ContainerStarted","Data":"47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da"} Nov 02 23:11:52.010262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.010217 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:52.163292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.163250 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:52.428004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.427977 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:52.514849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.514820 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\" (UniqueName: \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\") pod \"9edcd3d3-661d-4423-a151-75b803db33bf\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " Nov 02 23:11:52.515028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.515008 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\") pod \"9edcd3d3-661d-4423-a151-75b803db33bf\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " Nov 02 23:11:52.515079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.515050 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\") pod \"9edcd3d3-661d-4423-a151-75b803db33bf\" (UID: \"9edcd3d3-661d-4423-a151-75b803db33bf\") " Nov 02 23:11:52.515480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.515444 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca" pod "9edcd3d3-661d-4423-a151-75b803db33bf" (UID: "9edcd3d3-661d-4423-a151-75b803db33bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:52.515787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.515766 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:52.517250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.517219 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7" pod "9edcd3d3-661d-4423-a151-75b803db33bf" (UID: "9edcd3d3-661d-4423-a151-75b803db33bf"). InnerVolumeSpecName "kube-api-access-ws7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:52.517469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.517450 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca" pod "9edcd3d3-661d-4423-a151-75b803db33bf" (UID: "9edcd3d3-661d-4423-a151-75b803db33bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:52.617376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.617299 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws7g7\" (UniqueName: \"kubernetes.io/projected/9edcd3d3-661d-4423-a151-75b803db33bf-kube-api-access-ws7g7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:52.617376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:52.617329 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edcd3d3-661d-4423-a151-75b803db33bf-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:11:53.257101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.257069 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" Nov 02 23:11:53.257535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.257100 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk" event={"ID":"9edcd3d3-661d-4423-a151-75b803db33bf","Type":"ContainerDied","Data":"47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da"} Nov 02 23:11:53.257535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.257148 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47be4d042e9a0283ec0fd39cb8e6f928a12b62d4d1bfb2d55bce06f8358678da" Nov 02 23:11:53.921536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.920113 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:52504->10.133.0.104:15021: read: connection reset by peer" start-of-body= Nov 02 23:11:53.921536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.920224 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:52504->10.133.0.104:15021: read: connection reset by peer" Nov 02 23:11:53.958212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:53.958164 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:54.266111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.266016 4494 generic.go:356] "Generic (PLEG): container finished" podID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerID="d30bb4641382cbfbc16be521b4dc36a157a2a612c22ddb4a4b78eb987a2d8455" exitCode=0 Nov 02 23:11:54.266111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.266070 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerDied","Data":"d30bb4641382cbfbc16be521b4dc36a157a2a612c22ddb4a4b78eb987a2d8455"} Nov 02 23:11:54.266562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.266503 4494 scope.go:122] "RemoveContainer" containerID="d30bb4641382cbfbc16be521b4dc36a157a2a612c22ddb4a4b78eb987a2d8455" Nov 02 23:11:54.267915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.267889 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:11:54.267979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.267952 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:11:54.268014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:54.267985 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:11:55.276111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:55.276071 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerStarted","Data":"8db44ae5e8e08bd484bc59437356093ef0f827a382f30d3a17df6376e0dfe50b"} Nov 02 23:11:56.178375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:56.178336 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:11:56.178559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:56.178540 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:11:56.178614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:56.178595 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:11:57.177792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:57.177758 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:11:57.178265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:57.177824 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:11:58.073633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:58.073590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:58.177646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:58.177606 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:11:58.177837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:58.177668 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:11:58.226022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:58.225987 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:59.082534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:59.082501 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:11:59.194200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:59.180034 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:11:59.194200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:11:59.180154 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:12:00.177927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:00.177895 4494 patch_prober.go:29] interesting pod/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 23:12:00.178131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:00.177949 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 23:12:01.182326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:01.182296 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:12:01.182784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:01.182740 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:12:01.183617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:01.183598 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:12:01.662899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:01.662860 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:02.169761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.169727 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:12:02.170531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.170511 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9edcd3d3-661d-4423-a151-75b803db33bf" containerName="revoke-keys" Nov 02 23:12:02.170591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.170533 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edcd3d3-661d-4423-a151-75b803db33bf" containerName="revoke-keys" Nov 02 23:12:02.170649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.170640 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edcd3d3-661d-4423-a151-75b803db33bf" containerName="revoke-keys" Nov 02 23:12:02.173972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.173953 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.391515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.391467 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.434253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.430165 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.434684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.434660 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.624304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.624264 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2l2\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.624646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.624621 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.624827 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.632284 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.632329 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.632363 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.440359 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.407387 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8e34df-openshift-default-dockercfg-bjzfp\"" Nov 02 23:12:02.634214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.455693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:02.733463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733374 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733471 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733494 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2l2\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733528 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733557 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733584 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733612 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.733648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733645 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.733837 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.734010 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.734056 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.734214 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.734334 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.734569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.734542 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.736445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.736423 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.736562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.736548 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.796308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.796279 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:02.796564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:02.796541 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2l2\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\") pod \"e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:03.088770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.088687 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:03.139710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.139647 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:03.521533 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:03.521505 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f133f4_b997_4a5a_983f_4c52fddc641a.slice/crio-20142dec93fbf338226122e370afdbc8fd375469b6ace61af1338b4352c78ced WatchSource:0}: Error finding container 20142dec93fbf338226122e370afdbc8fd375469b6ace61af1338b4352c78ced: Status 404 returned error can't find the container with id 20142dec93fbf338226122e370afdbc8fd375469b6ace61af1338b4352c78ced Nov 02 23:12:03.522407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.522386 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:12:03.524212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.524146 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:03.524278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.524262 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:03.524318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.524302 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:03.580066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:03.580036 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerStarted","Data":"20142dec93fbf338226122e370afdbc8fd375469b6ace61af1338b4352c78ced"} Nov 02 23:12:04.589570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:04.589531 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerStarted","Data":"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538"} Nov 02 23:12:04.713234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:04.713205 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:12:04.715816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:04.715775 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podStartSLOduration=2.715762208 podStartE2EDuration="2.715762208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:04.711556728 +0000 UTC m=+1439.141426591" watchObservedRunningTime="2026-11-02 23:12:04.715762208 +0000 UTC m=+1439.145632068" Nov 02 23:12:04.741914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:04.741886 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:05.093580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:05.093550 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:05.168744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:05.168712 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:05.602263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:05.602227 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:05.603666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:05.603640 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:12:06.227609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:06.227581 4494 scope.go:122] "RemoveContainer" containerID="83bac4642b795c6e3fb7f52841530f0df66c6427957eeb9e74d36cd8d1cc10d6" Nov 02 23:12:06.326841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:06.326809 4494 scope.go:122] "RemoveContainer" containerID="a4d90a5032f3c450431b179fd25ef562fafbed4962090768882c85d524e9a504" Nov 02 23:12:06.424778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:06.420837 4494 scope.go:122] "RemoveContainer" containerID="cf762144c1fb77831e68c4e7dfc52b5519083f4273e9a3c5a9e24d89a16bde45" Nov 02 23:12:07.535607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:07.535574 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:07.742453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:07.742420 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:07.893710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:07.893628 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:09.685481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:09.685447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:10.216955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:10.216919 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:12.041997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.041969 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85"] Nov 02 23:12:12.048057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.048036 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.102880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.102854 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85"] Nov 02 23:12:12.116918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.116892 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.117193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.117094 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.117193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.117133 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7ng\" (UniqueName: \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.148643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.148613 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:12.148786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.148743 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:12:12.218494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.218453 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.218737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.218718 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.218846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.218767 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7ng\" (UniqueName: \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.219515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.219490 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.221522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.221501 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.281076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.281039 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7ng\" (UniqueName: \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\") pod \"maas-api-revoke-sub-94ac1bb6-2zp85\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.362446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.362376 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:12.740707 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:12.740670 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdfde9a_92e0_4f94_ab3d_7683942e8b7d.slice/crio-563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f WatchSource:0}: Error finding container 563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f: Status 404 returned error can't find the container with id 563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f Nov 02 23:12:12.741469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.741441 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85"] Nov 02 23:12:12.883033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:12.883002 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" event={"ID":"afdfde9a-92e0-4f94-ab3d-7683942e8b7d","Type":"ContainerStarted","Data":"563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f"} Nov 02 23:12:13.716917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:13.716881 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:13.892480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:13.892400 4494 generic.go:356] "Generic (PLEG): container finished" podID="afdfde9a-92e0-4f94-ab3d-7683942e8b7d" containerID="c685352ea401b316167eeae411bd5fa35ba7ee4b2356447b5d967d3f63b102e7" exitCode=0 Nov 02 23:12:13.892625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:13.892483 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" event={"ID":"afdfde9a-92e0-4f94-ab3d-7683942e8b7d","Type":"ContainerDied","Data":"c685352ea401b316167eeae411bd5fa35ba7ee4b2356447b5d967d3f63b102e7"} Nov 02 23:12:14.927036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:14.926923 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:15.201770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.201717 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:15.352440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.352401 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:15.450952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.450682 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\" (UniqueName: \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\") pod \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " Nov 02 23:12:15.450952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.450807 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\") pod \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " Nov 02 23:12:15.451256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.451100 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\") pod \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\" (UID: \"afdfde9a-92e0-4f94-ab3d-7683942e8b7d\") " Nov 02 23:12:15.475220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.462444 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca" pod "afdfde9a-92e0-4f94-ab3d-7683942e8b7d" (UID: "afdfde9a-92e0-4f94-ab3d-7683942e8b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:15.599232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.597579 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:12:15.600285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.599898 4494 cpu_manager.go:414] "containerMap: removing container" podUID="afdfde9a-92e0-4f94-ab3d-7683942e8b7d" containerName="revoke-keys" Nov 02 23:12:15.600285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.599927 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdfde9a-92e0-4f94-ab3d-7683942e8b7d" containerName="revoke-keys" Nov 02 23:12:15.600285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.600128 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdfde9a-92e0-4f94-ab3d-7683942e8b7d" containerName="revoke-keys" Nov 02 23:12:15.605041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.604950 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:15.606637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.605347 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.663564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.661911 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8e34df\"" Nov 02 23:12:15.663564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.662113 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8e34df-dockercfg-82tnr\"" Nov 02 23:12:15.731380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.731330 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng" pod "afdfde9a-92e0-4f94-ab3d-7683942e8b7d" (UID: "afdfde9a-92e0-4f94-ab3d-7683942e8b7d"). InnerVolumeSpecName "kube-api-access-jx7ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:15.750779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.731383 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca" pod "afdfde9a-92e0-4f94-ab3d-7683942e8b7d" (UID: "afdfde9a-92e0-4f94-ab3d-7683942e8b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:15.757466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.757429 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bvb\" (UniqueName: \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.774523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.774484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.775081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.775029 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7ng\" (UniqueName: \"kubernetes.io/projected/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-kube-api-access-jx7ng\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:15.775278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.775255 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdfde9a-92e0-4f94-ab3d-7683942e8b7d-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:15.886296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.885967 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.886296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.886192 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bvb\" (UniqueName: \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.887438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.886852 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:15.896368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.896335 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:12:15.908832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.908805 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" event={"ID":"afdfde9a-92e0-4f94-ab3d-7683942e8b7d","Type":"ContainerDied","Data":"563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f"} Nov 02 23:12:15.908934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.908849 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563c7b5b8e2f276038143c6e13235be9774bc26032f5e7587490b872cc565b9f" Nov 02 23:12:15.908934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.908816 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85" Nov 02 23:12:15.948186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:15.948134 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bvb\" (UniqueName: \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\") pod \"payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:16.039629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:16.039545 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:16.424900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:16.424872 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:12:16.917100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:16.917064 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" event={"ID":"2a09a9bc-900a-45a1-bf3a-8f75da4d1197","Type":"ContainerStarted","Data":"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045"} Nov 02 23:12:16.917328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:16.917142 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:16.917328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:16.917210 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" event={"ID":"2a09a9bc-900a-45a1-bf3a-8f75da4d1197","Type":"ContainerStarted","Data":"649bcdc59a650cc4aa8d4c853b6feaaf79403b62e905c421caeee2b521c8adc4"} Nov 02 23:12:17.037259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:17.037197 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" podStartSLOduration=2.0371667 podStartE2EDuration="2.0371667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:17.034358268 +0000 UTC m=+1451.464228137" watchObservedRunningTime="2026-11-02 23:12:17.0371667 +0000 UTC m=+1451.467036560" Nov 02 23:12:17.562120 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:17.562086 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:12:20.060628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:20.060585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:21.615919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:21.615867 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:24.931164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:24.931128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:25.088257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:25.088221 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:25.248505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:25.248419 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:26.306408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.306370 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699"] Nov 02 23:12:26.310735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.310718 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.361320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.361288 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6zj\" (UniqueName: \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.361549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.361476 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.361626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.361595 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.367618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.367590 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699"] Nov 02 23:12:26.412245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.412215 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:26.412424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.412309 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:12:26.480201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.480128 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6zj\" (UniqueName: \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.480591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.480567 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.480733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.480719 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.495195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.481876 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.521341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.507927 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.575358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.575246 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6zj\" (UniqueName: \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\") pod \"maas-api-revoke-sub-cf5db968-p9699\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:26.626559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:26.626523 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:27.058421 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:27.058393 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a67d767_3ca7_4aad_86c7_88b2f8372e61.slice/crio-eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402 WatchSource:0}: Error finding container eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402: Status 404 returned error can't find the container with id eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402 Nov 02 23:12:27.060070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:27.060037 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699"] Nov 02 23:12:27.929711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:27.929682 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:12:28.059040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:28.059007 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" event={"ID":"1a67d767-3ca7-4aad-86c7-88b2f8372e61","Type":"ContainerStarted","Data":"0f05f63008a220f74f574b69d6607d1abf9edbfc9f4259b6852de7d9984d8f82"} Nov 02 23:12:28.059210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:28.059058 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" event={"ID":"1a67d767-3ca7-4aad-86c7-88b2f8372e61","Type":"ContainerStarted","Data":"eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402"} Nov 02 23:12:28.183112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:28.183005 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" podStartSLOduration=2.182989712 podStartE2EDuration="2.182989712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:28.179697342 +0000 UTC m=+1462.609567207" watchObservedRunningTime="2026-11-02 23:12:28.182989712 +0000 UTC m=+1462.612859573" Nov 02 23:12:29.068061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:29.068028 4494 generic.go:356] "Generic (PLEG): container finished" podID="1a67d767-3ca7-4aad-86c7-88b2f8372e61" containerID="0f05f63008a220f74f574b69d6607d1abf9edbfc9f4259b6852de7d9984d8f82" exitCode=0 Nov 02 23:12:29.068494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:29.068096 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" event={"ID":"1a67d767-3ca7-4aad-86c7-88b2f8372e61","Type":"ContainerDied","Data":"0f05f63008a220f74f574b69d6607d1abf9edbfc9f4259b6852de7d9984d8f82"} Nov 02 23:12:30.055530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.055495 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:30.332545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.332508 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:30.351897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.351872 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\") pod \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " Nov 02 23:12:30.352077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.351927 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\" (UniqueName: \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\") pod \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " Nov 02 23:12:30.352141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.352115 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\") pod \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\" (UID: \"1a67d767-3ca7-4aad-86c7-88b2f8372e61\") " Nov 02 23:12:30.352358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.352315 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca" pod "1a67d767-3ca7-4aad-86c7-88b2f8372e61" (UID: "1a67d767-3ca7-4aad-86c7-88b2f8372e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:30.352731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.352712 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:30.354252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.354228 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj" pod "1a67d767-3ca7-4aad-86c7-88b2f8372e61" (UID: "1a67d767-3ca7-4aad-86c7-88b2f8372e61"). InnerVolumeSpecName "kube-api-access-2z6zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:30.354341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.354287 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca" pod "1a67d767-3ca7-4aad-86c7-88b2f8372e61" (UID: "1a67d767-3ca7-4aad-86c7-88b2f8372e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:30.453850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.453818 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6zj\" (UniqueName: \"kubernetes.io/projected/1a67d767-3ca7-4aad-86c7-88b2f8372e61-kube-api-access-2z6zj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:30.453850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:30.453845 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67d767-3ca7-4aad-86c7-88b2f8372e61-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:31.085452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:31.085421 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" event={"ID":"1a67d767-3ca7-4aad-86c7-88b2f8372e61","Type":"ContainerDied","Data":"eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402"} Nov 02 23:12:31.085452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:31.085449 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699" Nov 02 23:12:31.085663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:31.085470 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2d9893cf9af64358889128fd83c53951555d3ace5b88ad34c90c0d46f0e402" Nov 02 23:12:34.814451 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:34.814412 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-832363c1011a: secret "agc-cred-832363c1011a" not found Nov 02 23:12:34.814451 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:34.814445 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-91389874e6b5: secret "agc-cred-91389874e6b5" not found Nov 02 23:12:34.814968 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:34.814466 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj: [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:34.814968 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:34.814543 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials podName:e9325081-1096-49b3-a093-24a6916bd9af nodeName:}" failed. No retries permitted until 2026-11-02 23:12:35.314523049 +0000 UTC m=+1469.744392891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials") pod "payload-processing-external-model-5969b46dfd-d2ppj" (UID: "e9325081-1096-49b3-a093-24a6916bd9af") : [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:34.982450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:34.982417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:35.333862 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:35.333833 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-832363c1011a: secret "agc-cred-832363c1011a" not found Nov 02 23:12:35.333862 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:35.333858 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-91389874e6b5: secret "agc-cred-91389874e6b5" not found Nov 02 23:12:35.333862 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:35.333871 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj: [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:35.334250 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:35.333937 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials podName:e9325081-1096-49b3-a093-24a6916bd9af nodeName:}" failed. No retries permitted until 2026-11-02 23:12:36.333921622 +0000 UTC m=+1470.763791464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials") pod "payload-processing-external-model-5969b46dfd-d2ppj" (UID: "e9325081-1096-49b3-a093-24a6916bd9af") : [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:36.345982 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:36.345948 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-832363c1011a: secret "agc-cred-832363c1011a" not found Nov 02 23:12:36.345982 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:36.345975 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-91389874e6b5: secret "agc-cred-91389874e6b5" not found Nov 02 23:12:36.345982 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:36.345988 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj: [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:36.346485 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:36.346058 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials podName:e9325081-1096-49b3-a093-24a6916bd9af nodeName:}" failed. No retries permitted until 2026-11-02 23:12:38.34604354 +0000 UTC m=+1472.775913381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials") pod "payload-processing-external-model-5969b46dfd-d2ppj" (UID: "e9325081-1096-49b3-a093-24a6916bd9af") : [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:38.375241 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:38.375209 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-832363c1011a: secret "agc-cred-832363c1011a" not found Nov 02 23:12:38.375241 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:38.375236 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-91389874e6b5: secret "agc-cred-91389874e6b5" not found Nov 02 23:12:38.375241 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:38.375250 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj: [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:38.375823 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:38.375322 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials podName:e9325081-1096-49b3-a093-24a6916bd9af nodeName:}" failed. No retries permitted until 2026-11-02 23:12:42.375306955 +0000 UTC m=+1476.805176816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials") pod "payload-processing-external-model-5969b46dfd-d2ppj" (UID: "e9325081-1096-49b3-a093-24a6916bd9af") : [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found] Nov 02 23:12:38.674272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:38.674231 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:12:38.701461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:38.701392 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:38.904548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:38.904486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:40.459370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:40.459325 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:42.473317 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:42.473283 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-832363c1011a: secret "agc-cred-832363c1011a" not found Nov 02 23:12:42.473317 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:42.473309 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-91389874e6b5: secret "agc-cred-91389874e6b5" not found Nov 02 23:12:42.473317 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:42.473319 4494 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cf53d560958d: secret "agc-cred-cf53d560958d" not found Nov 02 23:12:42.473317 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:42.473328 4494 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj: [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found, secret "agc-cred-cf53d560958d" not found] Nov 02 23:12:42.473805 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:42.473388 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials podName:e9325081-1096-49b3-a093-24a6916bd9af nodeName:}" failed. No retries permitted until 2026-11-02 23:12:50.473371707 +0000 UTC m=+1484.903241546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials") pod "payload-processing-external-model-5969b46dfd-d2ppj" (UID: "e9325081-1096-49b3-a093-24a6916bd9af") : [secret "agc-cred-832363c1011a" not found, secret "agc-cred-91389874e6b5" not found, secret "agc-cred-cf53d560958d" not found] Nov 02 23:12:43.675374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:43.675339 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" podUID="e9325081-1096-49b3-a093-24a6916bd9af" containerName="payload-processing" containerID="cri-o://f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb" gracePeriod=25 Nov 02 23:12:43.942146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:43.942086 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:12:44.097000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.096965 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\") pod \"e9325081-1096-49b3-a093-24a6916bd9af\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " Nov 02 23:12:44.097198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097019 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\") pod \"e9325081-1096-49b3-a093-24a6916bd9af\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " Nov 02 23:12:44.097198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097157 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\") pod \"e9325081-1096-49b3-a093-24a6916bd9af\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " Nov 02 23:12:44.097290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097279 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\") pod \"e9325081-1096-49b3-a093-24a6916bd9af\" (UID: \"e9325081-1096-49b3-a093-24a6916bd9af\") " Nov 02 23:12:44.097494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097473 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume" pod "e9325081-1096-49b3-a093-24a6916bd9af" (UID: "e9325081-1096-49b3-a093-24a6916bd9af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:44.097649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097621 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume" pod "e9325081-1096-49b3-a093-24a6916bd9af" (UID: "e9325081-1096-49b3-a093-24a6916bd9af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:44.097905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097890 4494 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-routing-overlay-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.097960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.097909 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9325081-1096-49b3-a093-24a6916bd9af-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.099483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.099458 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm" pod "e9325081-1096-49b3-a093-24a6916bd9af" (UID: "e9325081-1096-49b3-a093-24a6916bd9af"). InnerVolumeSpecName "kube-api-access-l7tsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:44.099578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.099556 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials" pod "e9325081-1096-49b3-a093-24a6916bd9af" (UID: "e9325081-1096-49b3-a093-24a6916bd9af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:44.204088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.204000 4494 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-provider-credentials\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.204088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.204039 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tsm\" (UniqueName: \"kubernetes.io/projected/e9325081-1096-49b3-a093-24a6916bd9af-kube-api-access-l7tsm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:44.206746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.206717 4494 generic.go:356] "Generic (PLEG): container finished" podID="e9325081-1096-49b3-a093-24a6916bd9af" containerID="f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb" exitCode=0 Nov 02 23:12:44.206860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.206808 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" Nov 02 23:12:44.206860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.206810 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" event={"ID":"e9325081-1096-49b3-a093-24a6916bd9af","Type":"ContainerDied","Data":"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb"} Nov 02 23:12:44.207004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.206911 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj" event={"ID":"e9325081-1096-49b3-a093-24a6916bd9af","Type":"ContainerDied","Data":"d8d65610842a2afb1c3b1b3b7c4d0ce93645fb906ab97832873408c45278a2dd"} Nov 02 23:12:44.207004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.206978 4494 scope.go:122] "RemoveContainer" containerID="f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb" Nov 02 23:12:44.217606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.217589 4494 scope.go:122] "RemoveContainer" containerID="f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb" Nov 02 23:12:44.217856 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:44.217839 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb\": container with ID starting with f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb not found: ID does not exist" containerID="f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb" Nov 02 23:12:44.217912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.217863 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb"} err="failed to get container status \"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb\": rpc error: code = NotFound desc = could not find container \"f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb\": container with ID starting with f6be20c4fc676e4738f393a0fc8fa156c09a07e63a6fc153ccbe09db25fba1cb not found: ID does not exist" Nov 02 23:12:44.376227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.376195 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:12:44.380184 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:44.380152 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5969b46dfd-d2ppj"] Nov 02 23:12:45.374481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:45.374435 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:45.557825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:45.557793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:45.695996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:45.695968 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9325081-1096-49b3-a093-24a6916bd9af" path="/var/lib/kubelet/pods/e9325081-1096-49b3-a093-24a6916bd9af/volumes" Nov 02 23:12:45.702679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:45.702644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:46.670583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:46.670545 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:47.302205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.297329 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.314814 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1a67d767-3ca7-4aad-86c7-88b2f8372e61" containerName="revoke-keys" Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.314857 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a67d767-3ca7-4aad-86c7-88b2f8372e61" containerName="revoke-keys" Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.314896 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e9325081-1096-49b3-a093-24a6916bd9af" containerName="payload-processing" Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.314919 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9325081-1096-49b3-a093-24a6916bd9af" containerName="payload-processing" Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.315114 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a67d767-3ca7-4aad-86c7-88b2f8372e61" containerName="revoke-keys" Nov 02 23:12:47.317201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.315132 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9325081-1096-49b3-a093-24a6916bd9af" containerName="payload-processing" Nov 02 23:12:47.354636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.354305 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.378733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379621 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379681 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379741 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379776 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379803 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnblc\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379884 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379906 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379923 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.386197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.379946 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.428200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.415344 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5153fe-openshift-default-dockercfg-jmks5\"" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493000 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493056 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493091 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493191 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493264 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493350 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493400 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493433 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnblc\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.493557 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.501310 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.504874 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.505949 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.506771 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.507004 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.537964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.516765 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.554353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.554046 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:12:47.643200 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:12:47.638993 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:12:47.643200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.639552 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.669100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.665860 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.715234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.704497 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnblc\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\") pod \"e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.723470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.723439 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:47.991672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:47.991643 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:48.165393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:48.165359 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:12:48.200318 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:48.200290 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43399f3f_7d1a_468c_9037_a1e97fbed8fa.slice/crio-4380895973f71f337aa5961c1baddc43a0b32e22bcf609806914d8f92420e53b WatchSource:0}: Error finding container 4380895973f71f337aa5961c1baddc43a0b32e22bcf609806914d8f92420e53b: Status 404 returned error can't find the container with id 4380895973f71f337aa5961c1baddc43a0b32e22bcf609806914d8f92420e53b Nov 02 23:12:48.203089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:48.203052 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:48.203239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:48.203120 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:48.203239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:48.203149 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:12:48.242759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:48.242683 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerStarted","Data":"4380895973f71f337aa5961c1baddc43a0b32e22bcf609806914d8f92420e53b"} Nov 02 23:12:49.282198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:49.280455 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerStarted","Data":"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f"} Nov 02 23:12:49.460857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:49.460807 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podStartSLOduration=2.460792598 podStartE2EDuration="2.460792598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:49.459598724 +0000 UTC m=+1483.889468584" watchObservedRunningTime="2026-11-02 23:12:49.460792598 +0000 UTC m=+1483.890662460" Nov 02 23:12:49.724637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:49.724603 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:49.730653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:49.730631 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:50.287707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:50.287677 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:50.288687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:50.288665 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:12:50.876226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:50.876189 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:51.158361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.158273 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc"] Nov 02 23:12:51.163887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.163852 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.220838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.220811 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc"] Nov 02 23:12:51.244962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.244930 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.245153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.244991 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.245265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.245145 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsws\" (UniqueName: \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.265125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.265102 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:51.265262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.265148 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:12:51.346828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.346791 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsws\" (UniqueName: \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.347251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.347042 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.347251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.347116 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.347782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.347755 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.349557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.349534 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.408782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.408695 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsws\" (UniqueName: \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\") pod \"maas-api-revoke-sub-663c2edb-gwndc\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.478899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.478873 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:51.869365 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:51.869336 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9bad71_702a_48b4_bafc_2cf309a166ac.slice/crio-6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99 WatchSource:0}: Error finding container 6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99: Status 404 returned error can't find the container with id 6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99 Nov 02 23:12:51.870393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:51.870371 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc"] Nov 02 23:12:52.304508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.304462 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" event={"ID":"1e9bad71-702a-48b4-bafc-2cf309a166ac","Type":"ContainerStarted","Data":"16f9acc360a5685022ba52b0bbe59484d07ea44ede4be75e8c4b1c71f3dcf737"} Nov 02 23:12:52.304688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.304560 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" event={"ID":"1e9bad71-702a-48b4-bafc-2cf309a166ac","Type":"ContainerStarted","Data":"6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99"} Nov 02 23:12:52.412328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.412298 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:52.425380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.425335 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" podStartSLOduration=1.425321667 podStartE2EDuration="1.425321667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:52.422261538 +0000 UTC m=+1486.852131400" watchObservedRunningTime="2026-11-02 23:12:52.425321667 +0000 UTC m=+1486.855191528" Nov 02 23:12:52.581084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.581049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:52.734140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:52.734094 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:53.313920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:53.313888 4494 generic.go:356] "Generic (PLEG): container finished" podID="1e9bad71-702a-48b4-bafc-2cf309a166ac" containerID="16f9acc360a5685022ba52b0bbe59484d07ea44ede4be75e8c4b1c71f3dcf737" exitCode=0 Nov 02 23:12:53.314107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:53.313935 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" event={"ID":"1e9bad71-702a-48b4-bafc-2cf309a166ac","Type":"ContainerDied","Data":"16f9acc360a5685022ba52b0bbe59484d07ea44ede4be75e8c4b1c71f3dcf737"} Nov 02 23:12:54.475048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.475023 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:54.592106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.592010 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\") pod \"1e9bad71-702a-48b4-bafc-2cf309a166ac\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " Nov 02 23:12:54.592304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.592236 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\" (UniqueName: \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\") pod \"1e9bad71-702a-48b4-bafc-2cf309a166ac\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " Nov 02 23:12:54.592304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.592287 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\") pod \"1e9bad71-702a-48b4-bafc-2cf309a166ac\" (UID: \"1e9bad71-702a-48b4-bafc-2cf309a166ac\") " Nov 02 23:12:54.592748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.592714 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca" pod "1e9bad71-702a-48b4-bafc-2cf309a166ac" (UID: "1e9bad71-702a-48b4-bafc-2cf309a166ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:54.592971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.592922 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:54.594995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.594966 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca" pod "1e9bad71-702a-48b4-bafc-2cf309a166ac" (UID: "1e9bad71-702a-48b4-bafc-2cf309a166ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:54.595124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.595007 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws" pod "1e9bad71-702a-48b4-bafc-2cf309a166ac" (UID: "1e9bad71-702a-48b4-bafc-2cf309a166ac"). InnerVolumeSpecName "kube-api-access-5fsws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:54.694654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.694620 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9bad71-702a-48b4-bafc-2cf309a166ac-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:54.694654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:54.694651 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsws\" (UniqueName: \"kubernetes.io/projected/1e9bad71-702a-48b4-bafc-2cf309a166ac-kube-api-access-5fsws\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:12:55.329212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:55.329183 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" Nov 02 23:12:55.329395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:55.329210 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc" event={"ID":"1e9bad71-702a-48b4-bafc-2cf309a166ac","Type":"ContainerDied","Data":"6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99"} Nov 02 23:12:55.329395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:55.329260 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6acda1a70e1e4729159757bfc70951ace5304456a6c01133318515032a683f99" Nov 02 23:12:55.752747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:55.752716 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:55.982198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:55.981684 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:56.217334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.216086 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:12:56.217334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.217276 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1e9bad71-702a-48b4-bafc-2cf309a166ac" containerName="revoke-keys" Nov 02 23:12:56.217334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.217299 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9bad71-702a-48b4-bafc-2cf309a166ac" containerName="revoke-keys" Nov 02 23:12:56.217639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.217506 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9bad71-702a-48b4-bafc-2cf309a166ac" containerName="revoke-keys" Nov 02 23:12:56.224221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.222616 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.275209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.270902 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dpl\" (UniqueName: \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.277561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.277529 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.284193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.280535 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5153fe\"" Nov 02 23:12:56.284193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.281564 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5153fe-dockercfg-qggrn\"" Nov 02 23:12:56.304107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.304053 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:12:56.401195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.393275 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dpl\" (UniqueName: \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.446855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.436336 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.446855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.437586 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.518368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.514668 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dpl\" (UniqueName: \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\") pod \"payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.552709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.550565 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:56.633217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:56.631079 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:12:57.148545 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:12:57.148514 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3539f21f_6d47_4226_8da5_e7d28b270bd2.slice/crio-34404673f662fa2a10749686b55c87b8abe3286d024bc1ed7ac94d3c58a1fd68 WatchSource:0}: Error finding container 34404673f662fa2a10749686b55c87b8abe3286d024bc1ed7ac94d3c58a1fd68: Status 404 returned error can't find the container with id 34404673f662fa2a10749686b55c87b8abe3286d024bc1ed7ac94d3c58a1fd68 Nov 02 23:12:57.148940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:57.148920 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:12:57.457235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:57.457193 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" event={"ID":"3539f21f-6d47-4226-8da5-e7d28b270bd2","Type":"ContainerStarted","Data":"4bd52c4d142fab129bef441aff5454c3216321613623b913978c2d2885b7090a"} Nov 02 23:12:57.457395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:57.457281 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:12:57.457395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:57.457306 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" event={"ID":"3539f21f-6d47-4226-8da5-e7d28b270bd2","Type":"ContainerStarted","Data":"34404673f662fa2a10749686b55c87b8abe3286d024bc1ed7ac94d3c58a1fd68"} Nov 02 23:12:57.580047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:12:57.579993 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" podStartSLOduration=1.579979923 podStartE2EDuration="1.579979923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:57.574320362 +0000 UTC m=+1492.004190236" watchObservedRunningTime="2026-11-02 23:12:57.579979923 +0000 UTC m=+1492.009849827" Nov 02 23:13:00.344948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:00.344919 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:13:02.065811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:02.065772 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:02.528806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:02.528770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:03.789197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:03.785605 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:04.282204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:04.281799 4494 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:04.581567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:04.580454 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:06.760252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:06.760218 4494 scope.go:122] "RemoveContainer" containerID="4d7467f0b9c59e43672a8f3d7871627565057def745377c63c02c2bd413bcec2" Nov 02 23:13:07.771620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:07.771575 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:07.940069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:07.930641 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:08.467251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:08.467223 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:13:11.664298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.664264 4494 generic.go:356] "Generic (PLEG): container finished" podID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerID="fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538" exitCode=0 Nov 02 23:13:11.664816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.664346 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerDied","Data":"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538"} Nov 02 23:13:11.664914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.664879 4494 scope.go:122] "RemoveContainer" containerID="fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538" Nov 02 23:13:11.666502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.666470 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:11.666624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.666560 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:11.666624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:11.666607 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:12.674635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:12.674604 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerStarted","Data":"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5"} Nov 02 23:13:12.804199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:12.803234 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:13.089459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:13.089371 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:13:13.089625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:13.089556 4494 patch_prober.go:29] interesting pod/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:13:13.089625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:13.089597 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 23:13:14.089894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:14.089858 4494 patch_prober.go:29] interesting pod/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:13:14.090494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:14.089927 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 23:13:15.089907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:15.089875 4494 patch_prober.go:29] interesting pod/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:13:15.090370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:15.089930 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 23:13:16.089286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:16.089245 4494 patch_prober.go:29] interesting pod/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:13:16.089506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:16.089310 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 23:13:17.071712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:17.071680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:17.089410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:17.089380 4494 patch_prober.go:29] interesting pod/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 23:13:17.089565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:17.089442 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 23:13:17.617000 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:13:17.616895 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:13:17.944782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:17.944749 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:18.188534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:18.188498 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:13:18.191106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:18.191079 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:13:18.196080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:18.196005 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:13:22.856885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:22.856798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:28.144440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:28.144394 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:33.184798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:33.184765 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:33.353129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:33.341529 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:33.494388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:33.494293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:34.035695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:34.035664 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:34.718474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:34.718438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:35.217763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.217730 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:13:35.222142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.222124 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.285255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.283656 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8ab111-openshift-default-dockercfg-5ppwc\"" Nov 02 23:13:35.308584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.308527 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.308971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.308947 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.312943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.312919 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.313234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.313210 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.316415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.316393 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.316668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.316652 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.318778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.316888 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.319262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.319242 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.319559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.319501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbh4s\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.416817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.416733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:35.428223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.419968 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.455980 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456083 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456115 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456274 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbh4s\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456318 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456376 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456650 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456688 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.456772 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.457370 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.457686 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.478325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.457915 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.558197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.511889 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.558197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.512604 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.647088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.647016 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbh4s\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.668250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.659824 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.714202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.708766 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.754100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.753990 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\") pod \"e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:35.837151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:35.837119 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:36.013899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.013712 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:36.315814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.315775 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:13:36.350787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.338048 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:36.350787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.338162 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:36.350787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.338302 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:13:36.887912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.887873 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerStarted","Data":"2b5bb72e6da84bfd9a046bb0c506d404cd42f6dbca23b770245defda35771ae0"} Nov 02 23:13:36.888496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:36.887934 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerStarted","Data":"f46b8dc6181949ab98990774e425f6a5f7e2250e7a7682934380a5accef9f08b"} Nov 02 23:13:37.012552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.012500 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podStartSLOduration=2.01248555 podStartE2EDuration="2.01248555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:37.008838994 +0000 UTC m=+1531.438708855" watchObservedRunningTime="2026-11-02 23:13:37.01248555 +0000 UTC m=+1531.442355487" Nov 02 23:13:37.570228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.570151 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:13:37.837725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.837631 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:37.842918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.842896 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:37.895296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.895269 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:37.896249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:37.896229 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:13:39.119472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:39.119437 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:40.319789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:40.319756 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:40.493126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:40.493076 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:40.649271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:40.649159 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:43.500474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:43.500435 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:43.712213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:43.704675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:44.010325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.000085 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:13:44.020855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.020828 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.088229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.043876 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:13:44.151779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.151654 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8ab111-dockercfg-v6j9z\"" Nov 02 23:13:44.151779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.151745 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8ab111\"" Nov 02 23:13:44.206666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.206622 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:13:44.263292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.239644 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ghn\" (UniqueName: \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.263731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.263683 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.296515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.296475 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.303892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.303721 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:13:44.366032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.365985 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ghn\" (UniqueName: \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.366231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.366100 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.366303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.366242 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.366303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.366295 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6b7\" (UniqueName: \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.366858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.366833 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.427771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.427744 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ghn\" (UniqueName: \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\") pod \"payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.447714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.447685 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:44.467310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.467283 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.467408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.467345 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6b7\" (UniqueName: \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.468005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.467978 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.531636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.531164 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6b7\" (UniqueName: \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\") pod \"payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.633107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.633076 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:44.826313 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:13:44.826283 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7651354_c457_4bd4_adec_2812a16af7ac.slice/crio-fc0eba4243f1ae1fef7a077b6306d66d8e2048e2c339980f352c6bde143d8c54 WatchSource:0}: Error finding container fc0eba4243f1ae1fef7a077b6306d66d8e2048e2c339980f352c6bde143d8c54: Status 404 returned error can't find the container with id fc0eba4243f1ae1fef7a077b6306d66d8e2048e2c339980f352c6bde143d8c54 Nov 02 23:13:44.827481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.827458 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:13:44.982331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:44.982281 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:45.060220 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:13:45.060156 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8972bd_107f_4acf_9337_3acd07e4b553.slice/crio-375eec72319102380e4f898152c007be68d2dbde71d852775e1e58e4334bb177 WatchSource:0}: Error finding container 375eec72319102380e4f898152c007be68d2dbde71d852775e1e58e4334bb177: Status 404 returned error can't find the container with id 375eec72319102380e4f898152c007be68d2dbde71d852775e1e58e4334bb177 Nov 02 23:13:45.078202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:45.078123 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:13:45.091803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:45.091747 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" event={"ID":"d7651354-c457-4bd4-adec-2812a16af7ac","Type":"ContainerStarted","Data":"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb"} Nov 02 23:13:45.092108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:45.092081 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" event={"ID":"d7651354-c457-4bd4-adec-2812a16af7ac","Type":"ContainerStarted","Data":"fc0eba4243f1ae1fef7a077b6306d66d8e2048e2c339980f352c6bde143d8c54"} Nov 02 23:13:45.093605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:45.093563 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:13:45.214230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:45.214164 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" podStartSLOduration=2.214149475 podStartE2EDuration="2.214149475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:45.209837424 +0000 UTC m=+1539.639707306" watchObservedRunningTime="2026-11-02 23:13:45.214149475 +0000 UTC m=+1539.644019336" Nov 02 23:13:46.100750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:46.100707 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" event={"ID":"1e8972bd-107f-4acf-9337-3acd07e4b553","Type":"ContainerStarted","Data":"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a"} Nov 02 23:13:46.101237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:46.100893 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:46.101237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:46.100926 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" event={"ID":"1e8972bd-107f-4acf-9337-3acd07e4b553","Type":"ContainerStarted","Data":"375eec72319102380e4f898152c007be68d2dbde71d852775e1e58e4334bb177"} Nov 02 23:13:46.219680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:46.219629 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" podStartSLOduration=2.219615539 podStartE2EDuration="2.219615539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:46.217603215 +0000 UTC m=+1540.647473100" watchObservedRunningTime="2026-11-02 23:13:46.219615539 +0000 UTC m=+1540.649485400" Nov 02 23:13:47.562526 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:13:47.562478 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:13:50.055003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:50.054953 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:51.702295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:51.702264 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:51.849526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:51.849478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:55.068065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:55.068029 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:56.111405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:56.111371 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:13:57.112770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:57.112742 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:13:57.113255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:13:57.112812 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:14:00.197818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:00.197781 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:03.050948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:03.050914 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5"] Nov 02 23:14:03.099695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:03.099660 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44387080-8g4p5"] Nov 02 23:14:03.702275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:03.702237 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e07e292-be6c-4752-8e1e-d7204949ddb6" path="/var/lib/kubelet/pods/2e07e292-be6c-4752-8e1e-d7204949ddb6/volumes" Nov 02 23:14:04.136815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.136731 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:04.361966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.361927 4494 generic.go:356] "Generic (PLEG): container finished" podID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerID="d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f" exitCode=0 Nov 02 23:14:04.362277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.362023 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerDied","Data":"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f"} Nov 02 23:14:04.362703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.362681 4494 scope.go:122] "RemoveContainer" containerID="d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f" Nov 02 23:14:04.364360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.364332 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:04.364441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.364397 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:04.364441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:04.364430 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:05.285354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.285318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:05.288236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.288214 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:14:05.423225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.423187 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerStarted","Data":"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71"} Nov 02 23:14:05.442032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.441987 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:05.723977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.723947 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:14:05.725273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.725244 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:05.725386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:05.725300 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:06.725003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:06.724969 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:06.725544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:06.725029 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:07.724527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:07.724494 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:07.724734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:07.724543 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:08.724807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:08.724771 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:08.725308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:08.724830 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:09.124744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:09.124643 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:14:09.725001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:09.724964 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:09.725450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:09.725017 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:10.593566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:10.593524 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:10.730344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:10.730312 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 23:14:10.737191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:10.737131 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 23:14:11.728460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:11.728431 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:14:11.728911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:11.728889 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:14:11.729535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:11.729516 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:14:15.049336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:15.049300 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz"] Nov 02 23:14:15.099820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:15.099789 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649c2fa9-4n4mz"] Nov 02 23:14:15.547689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:15.547654 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:15.700542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:15.700510 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abcfa9a-1356-4212-910b-36fcb4087597" path="/var/lib/kubelet/pods/9abcfa9a-1356-4212-910b-36fcb4087597/volumes" Nov 02 23:14:16.993587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:16.993558 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn"] Nov 02 23:14:16.999167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:16.999147 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.055510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.055482 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn"] Nov 02 23:14:17.069061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.069034 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.069250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.069193 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.069447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.069427 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkmx\" (UniqueName: \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.099777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.099740 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:14:17.099929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.099790 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:14:17.171255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.171224 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.171428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.171312 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.171487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.171457 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkmx\" (UniqueName: \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.172012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.171984 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.173961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.173939 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.233408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.233378 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkmx\" (UniqueName: \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\") pod \"maas-api-revoke-keys-e2e-mt-a7a67875-ssswn\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.313414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.313327 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:17.567292 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:14:17.567148 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:14:17.696676 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:14:17.696641 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24f22a1_cb47_4c97_aab3_338d60c124f0.slice/crio-d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f WatchSource:0}: Error finding container d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f: Status 404 returned error can't find the container with id d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f Nov 02 23:14:17.698052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:17.698028 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn"] Nov 02 23:14:18.049504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:18.049475 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb"] Nov 02 23:14:18.099152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:18.099120 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed8334a-gc9bb"] Nov 02 23:14:18.594896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:18.594842 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" event={"ID":"d24f22a1-cb47-4c97-aab3-338d60c124f0","Type":"ContainerStarted","Data":"e7f2d3d6c9b81b5bce397bbfcd4b0659133cbfa997b9330b197bdf47ea64f5f7"} Nov 02 23:14:18.594896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:18.594904 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" event={"ID":"d24f22a1-cb47-4c97-aab3-338d60c124f0","Type":"ContainerStarted","Data":"d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f"} Nov 02 23:14:18.713462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:18.713402 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" podStartSLOduration=2.7133875229999997 podStartE2EDuration="2.713387523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:18.712281406 +0000 UTC m=+1573.142151275" watchObservedRunningTime="2026-11-02 23:14:18.713387523 +0000 UTC m=+1573.143257649" Nov 02 23:14:19.604434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:19.604404 4494 generic.go:356] "Generic (PLEG): container finished" podID="d24f22a1-cb47-4c97-aab3-338d60c124f0" containerID="e7f2d3d6c9b81b5bce397bbfcd4b0659133cbfa997b9330b197bdf47ea64f5f7" exitCode=0 Nov 02 23:14:19.604956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:19.604505 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" event={"ID":"d24f22a1-cb47-4c97-aab3-338d60c124f0","Type":"ContainerDied","Data":"e7f2d3d6c9b81b5bce397bbfcd4b0659133cbfa997b9330b197bdf47ea64f5f7"} Nov 02 23:14:19.694951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:19.694906 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d4a83f-b177-42d7-af43-c1f57e906089" path="/var/lib/kubelet/pods/86d4a83f-b177-42d7-af43-c1f57e906089/volumes" Nov 02 23:14:20.766298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.766274 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:20.820585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.820556 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\") pod \"d24f22a1-cb47-4c97-aab3-338d60c124f0\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " Nov 02 23:14:20.820772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.820630 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\") pod \"d24f22a1-cb47-4c97-aab3-338d60c124f0\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " Nov 02 23:14:20.820772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.820675 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\" (UniqueName: \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\") pod \"d24f22a1-cb47-4c97-aab3-338d60c124f0\" (UID: \"d24f22a1-cb47-4c97-aab3-338d60c124f0\") " Nov 02 23:14:20.821060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.821035 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca" pod "d24f22a1-cb47-4c97-aab3-338d60c124f0" (UID: "d24f22a1-cb47-4c97-aab3-338d60c124f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:20.821784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.821760 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:20.822951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.822929 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca" pod "d24f22a1-cb47-4c97-aab3-338d60c124f0" (UID: "d24f22a1-cb47-4c97-aab3-338d60c124f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:14:20.823056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.823039 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx" pod "d24f22a1-cb47-4c97-aab3-338d60c124f0" (UID: "d24f22a1-cb47-4c97-aab3-338d60c124f0"). InnerVolumeSpecName "kube-api-access-fqkmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:20.852820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.852796 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:20.990975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.952195 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f22a1-cb47-4c97-aab3-338d60c124f0-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:20.990975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:20.952235 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqkmx\" (UniqueName: \"kubernetes.io/projected/d24f22a1-cb47-4c97-aab3-338d60c124f0-kube-api-access-fqkmx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:21.620533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:21.620499 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" Nov 02 23:14:21.620711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:21.620533 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn" event={"ID":"d24f22a1-cb47-4c97-aab3-338d60c124f0","Type":"ContainerDied","Data":"d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f"} Nov 02 23:14:21.620711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:21.620621 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d365f0a1616aa0da3948ac253db6a3670567291d0e07075932e10b4eff47d59f" Nov 02 23:14:26.025869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:26.025832 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:26.184451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:26.184418 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:26.390154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:26.390027 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:29.009680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:29.009646 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:29.751898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:29.751851 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:29.969070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:29.969030 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:14:30.097271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.095233 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn"] Nov 02 23:14:30.117358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.117291 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0251797-wk8sn"] Nov 02 23:14:30.428543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.428428 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:30.439486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.439424 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:14:30.440999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.440969 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d24f22a1-cb47-4c97-aab3-338d60c124f0" containerName="revoke-keys" Nov 02 23:14:30.441212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.441156 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24f22a1-cb47-4c97-aab3-338d60c124f0" containerName="revoke-keys" Nov 02 23:14:30.441568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.441548 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24f22a1-cb47-4c97-aab3-338d60c124f0" containerName="revoke-keys" Nov 02 23:14:30.472042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.472009 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.564317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.553816 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:14:30.579483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.565520 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlthq\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.579881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.579848 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580028 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580204 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580377 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580690 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580841 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.580966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.580949 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.581098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.581083 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.593374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.593338 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-62ab55-openshift-default-dockercfg-q94s6\"" Nov 02 23:14:30.629939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.629429 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:30.683159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.683073 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.710381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.710349 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.711368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.711227 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.711638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.711618 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.713209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.712634 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.713511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.712556 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.714354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.713051 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.714778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.714757 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlthq\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.715387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.715366 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.715561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.715543 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.716146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.716025 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.716963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.716945 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.717657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.717133 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.718076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.718054 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.723305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.723256 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.786240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.769848 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.801298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.801084 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlthq\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.803193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.801972 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:30.812262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.812058 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\") pod \"e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:30.884597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:30.884349 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:31.326256 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:14:31.326222 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d95a30d_fe5c_4f3b_81b7_e9b399516451.slice/crio-c4a7e33146a07535496b31e4f4e27f90b4e3b8d5d1368ec899c23fea31d5ef0f WatchSource:0}: Error finding container c4a7e33146a07535496b31e4f4e27f90b4e3b8d5d1368ec899c23fea31d5ef0f: Status 404 returned error can't find the container with id c4a7e33146a07535496b31e4f4e27f90b4e3b8d5d1368ec899c23fea31d5ef0f Nov 02 23:14:31.327207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.327163 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:14:31.328963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.328929 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:31.329034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.328998 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:31.329034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.329028 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:31.710922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.709685 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fdc364-a112-48f2-94eb-c005f3ebfe7f" path="/var/lib/kubelet/pods/c7fdc364-a112-48f2-94eb-c005f3ebfe7f/volumes" Nov 02 23:14:31.723888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.723854 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerStarted","Data":"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10"} Nov 02 23:14:31.724019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.723902 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerStarted","Data":"c4a7e33146a07535496b31e4f4e27f90b4e3b8d5d1368ec899c23fea31d5ef0f"} Nov 02 23:14:31.885428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:31.885391 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:32.031868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:32.031782 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:32.855847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:32.855815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:32.890288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:32.890263 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:33.067983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.067930 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podStartSLOduration=3.067913728 podStartE2EDuration="3.067913728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:31.881297493 +0000 UTC m=+1586.311167354" watchObservedRunningTime="2026-11-02 23:14:33.067913728 +0000 UTC m=+1587.497783620" Nov 02 23:14:33.077213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.077166 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm"] Nov 02 23:14:33.081109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.081089 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f09147-6ncnm"] Nov 02 23:14:33.408430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.408387 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:14:33.432925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.432893 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.589454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.560433 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:14:33.589454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.561403 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bf51d6-openshift-default-dockercfg-jm6p4\"" Nov 02 23:14:33.627802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.627756 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653516 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653680 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcptj\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653734 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653764 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653813 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653847 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653924 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.661595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.653980 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.724878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.724840 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e5ee60-55dc-47cb-83df-97cc93c2607b" path="/var/lib/kubelet/pods/b8e5ee60-55dc-47cb-83df-97cc93c2607b/volumes" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755271 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755356 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755518 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755584 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755762 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcptj\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755837 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755881 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755941 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.755996 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.761267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.757218 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.762031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.761333 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.762031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.761689 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.762031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.761985 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.770767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.770736 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.771525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.771473 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.809317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.786914 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.866264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.845663 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.893764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.893730 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:33.895330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.895301 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:14:33.904356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.904331 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcptj\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\") pod \"e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:33.989741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:33.989654 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:34.166200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.151321 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:34.450428 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:14:34.450398 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3407dc7f_387d_4f50_950b_fa841a010942.slice/crio-f3a812b29e611f4839eb2b1d415dab16c459e1da4439ad6fb5b0775904d5871a WatchSource:0}: Error finding container f3a812b29e611f4839eb2b1d415dab16c459e1da4439ad6fb5b0775904d5871a: Status 404 returned error can't find the container with id f3a812b29e611f4839eb2b1d415dab16c459e1da4439ad6fb5b0775904d5871a Nov 02 23:14:34.451563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.451541 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:14:34.453582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.453551 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:34.453652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.453616 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:34.453652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.453645 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:14:34.735535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.735427 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:34.898980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.898939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:34.902924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.902895 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerStarted","Data":"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3"} Nov 02 23:14:34.903062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.902956 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerStarted","Data":"f3a812b29e611f4839eb2b1d415dab16c459e1da4439ad6fb5b0775904d5871a"} Nov 02 23:14:34.969665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.969627 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" containerID="cri-o://1fb56813ae3836294fc6d66ddb89d79fc60cf43072f24a9a4946d05bcb83a88f" gracePeriod=25 Nov 02 23:14:34.990383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:34.990317 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:35.028497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:35.028446 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podStartSLOduration=2.028432544 podStartE2EDuration="2.028432544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:35.023916929 +0000 UTC m=+1589.453786792" watchObservedRunningTime="2026-11-02 23:14:35.028432544 +0000 UTC m=+1589.458302405" Nov 02 23:14:35.065401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:35.065370 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:35.996021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:35.995985 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:36.917927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:36.917896 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:36.918917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:36.918894 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:14:37.466583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:37.466553 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:38.538303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:38.538272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:38.818256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:38.812562 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:39.006197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:38.993344 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:14:39.025805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.025737 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.037913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.037860 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:14:39.171257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.143320 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.283218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.281918 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5tv\" (UniqueName: \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.283218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.155647 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-62ab55-dockercfg-vfq49\"" Nov 02 23:14:39.283218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.155280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:39.283218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.155691 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-62ab55\"" Nov 02 23:14:39.283218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.228329 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:14:39.365214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.361499 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.385388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.385356 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.385762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.385738 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5tv\" (UniqueName: \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.385989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.385975 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.386382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.386357 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:14:39.387739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.386658 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjcn\" (UniqueName: \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.387841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.387328 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.430748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.430670 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:39.450981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.450950 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5tv\" (UniqueName: \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\") pod \"payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.489160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.489130 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.489442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.489421 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjcn\" (UniqueName: \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.489780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.489762 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.551715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.551688 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjcn\" (UniqueName: \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\") pod \"payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:39.671443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.671414 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:39.687005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:39.686980 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:40.070467 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:14:40.070414 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c505e8_3891_4fb9_8af8_35347d043298.slice/crio-4e34f8484fc804d0f81bf7800a86cf5108412c2e7a9048a6d13ec2a59138184e WatchSource:0}: Error finding container 4e34f8484fc804d0f81bf7800a86cf5108412c2e7a9048a6d13ec2a59138184e: Status 404 returned error can't find the container with id 4e34f8484fc804d0f81bf7800a86cf5108412c2e7a9048a6d13ec2a59138184e Nov 02 23:14:40.071121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.071049 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:14:40.111053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.111022 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:14:40.119571 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:14:40.119526 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b92d4a_7076_4938_9911_d6baef3d917f.slice/crio-4aeb115318f23be6b1848e0cad8e349c242769de7e8dc6ce2e3fc2266582d93a WatchSource:0}: Error finding container 4aeb115318f23be6b1848e0cad8e349c242769de7e8dc6ce2e3fc2266582d93a: Status 404 returned error can't find the container with id 4aeb115318f23be6b1848e0cad8e349c242769de7e8dc6ce2e3fc2266582d93a Nov 02 23:14:40.997607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.997570 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" event={"ID":"d9b92d4a-7076-4938-9911-d6baef3d917f","Type":"ContainerStarted","Data":"511bbe99ad79f9c5a2084b1a32d8f167043c8541b609516d50cf3e1bbf6a5669"} Nov 02 23:14:40.998067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.997624 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" event={"ID":"d9b92d4a-7076-4938-9911-d6baef3d917f","Type":"ContainerStarted","Data":"4aeb115318f23be6b1848e0cad8e349c242769de7e8dc6ce2e3fc2266582d93a"} Nov 02 23:14:40.998067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.997664 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:40.999238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.999211 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" event={"ID":"f8c505e8-3891-4fb9-8af8-35347d043298","Type":"ContainerStarted","Data":"72df31f4daeb249c989e23f28808d7e59b9f45335bc7012dcc4b6a815181526e"} Nov 02 23:14:40.999362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.999253 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" event={"ID":"f8c505e8-3891-4fb9-8af8-35347d043298","Type":"ContainerStarted","Data":"4e34f8484fc804d0f81bf7800a86cf5108412c2e7a9048a6d13ec2a59138184e"} Nov 02 23:14:40.999362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:40.999276 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:41.116341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:41.116288 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" podStartSLOduration=2.116273258 podStartE2EDuration="2.116273258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:41.114216783 +0000 UTC m=+1595.544086635" watchObservedRunningTime="2026-11-02 23:14:41.116273258 +0000 UTC m=+1595.546143124" Nov 02 23:14:41.438195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:41.438141 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:14:41.984103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:41.984032 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" podStartSLOduration=3.984009603 podStartE2EDuration="3.984009603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:41.252405191 +0000 UTC m=+1595.682275085" watchObservedRunningTime="2026-11-02 23:14:41.984009603 +0000 UTC m=+1596.413879472" Nov 02 23:14:41.999820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:41.999780 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:14:43.187956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:43.187922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:43.343569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:43.343535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:43.617501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:43.617420 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:45.075858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:45.075820 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7"] Nov 02 23:14:45.121547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:45.121497 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6057cb0a-wwlx7"] Nov 02 23:14:45.695276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:45.695237 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e657ebe0-e50a-47da-99a5-c7b208286dbf" path="/var/lib/kubelet/pods/e657ebe0-e50a-47da-99a5-c7b208286dbf/volumes" Nov 02 23:14:46.049751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:46.049664 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w"] Nov 02 23:14:46.099831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:46.099801 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2836708d-bmj4w"] Nov 02 23:14:47.563195 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:14:47.563149 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:14:47.698839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:47.698804 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521e1ed0-51e4-47de-a993-29815ae80686" path="/var/lib/kubelet/pods/521e1ed0-51e4-47de-a993-29815ae80686/volumes" Nov 02 23:14:48.269900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:48.269872 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:51.433375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:51.433335 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:14:52.016783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:52.016752 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:14:52.016984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:52.016818 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:14:53.232080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:53.232043 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:14:55.166287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.166247 4494 generic.go:356] "Generic (PLEG): container finished" podID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerID="1fb56813ae3836294fc6d66ddb89d79fc60cf43072f24a9a4946d05bcb83a88f" exitCode=0 Nov 02 23:14:55.166744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.166315 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" event={"ID":"72d846c8-c350-4748-bde1-9d8e593c58d5","Type":"ContainerDied","Data":"1fb56813ae3836294fc6d66ddb89d79fc60cf43072f24a9a4946d05bcb83a88f"} Nov 02 23:14:55.252511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.252486 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:14:55.375967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.375934 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\") pod \"72d846c8-c350-4748-bde1-9d8e593c58d5\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " Nov 02 23:14:55.376159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.376070 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\" (UniqueName: \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\") pod \"72d846c8-c350-4748-bde1-9d8e593c58d5\" (UID: \"72d846c8-c350-4748-bde1-9d8e593c58d5\") " Nov 02 23:14:55.376354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.376331 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume" pod "72d846c8-c350-4748-bde1-9d8e593c58d5" (UID: "72d846c8-c350-4748-bde1-9d8e593c58d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:55.376716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.376697 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d846c8-c350-4748-bde1-9d8e593c58d5-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:55.378542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.378509 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz" pod "72d846c8-c350-4748-bde1-9d8e593c58d5" (UID: "72d846c8-c350-4748-bde1-9d8e593c58d5"). InnerVolumeSpecName "kube-api-access-bbmhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:55.478363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:55.478316 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmhz\" (UniqueName: \"kubernetes.io/projected/72d846c8-c350-4748-bde1-9d8e593c58d5-kube-api-access-bbmhz\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:14:56.175427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:56.175391 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" Nov 02 23:14:56.175427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:56.175414 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7" event={"ID":"72d846c8-c350-4748-bde1-9d8e593c58d5","Type":"ContainerDied","Data":"0a795c38d7818fd8495b6cd5b163a4c04950d8969b3e492d1ce0e48671c3b195"} Nov 02 23:14:56.176019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:56.175473 4494 scope.go:122] "RemoveContainer" containerID="1fb56813ae3836294fc6d66ddb89d79fc60cf43072f24a9a4946d05bcb83a88f" Nov 02 23:14:56.346247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:56.346206 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:14:56.397116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:56.397082 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-6m9k7"] Nov 02 23:14:57.695917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:57.695887 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" path="/var/lib/kubelet/pods/72d846c8-c350-4748-bde1-9d8e593c58d5/volumes" Nov 02 23:14:58.247830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:14:58.247798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:00.175282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.175243 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:00.176335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.176315 4494 cpu_manager.go:414] "containerMap: removing container" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" Nov 02 23:15:00.176436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.176337 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" Nov 02 23:15:00.176552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.176540 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d846c8-c350-4748-bde1-9d8e593c58d5" containerName="payload-processing" Nov 02 23:15:00.181200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.181154 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:00.181283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.181260 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.186602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.186580 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:00.186741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.186688 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.190813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.190794 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.226592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.226568 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:00.232747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.232727 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.232835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.232777 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:00.234283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.234261 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:15:00.237108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.237091 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.237591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.237572 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:00.242348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.242328 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.260684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.260659 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.260923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.260909 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vmj\" (UniqueName: \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.277907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.277877 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:00.283051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.283021 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:00.287893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.287865 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:00.324250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.324217 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:00.333875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.333848 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:00.362654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362625 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vmj\" (UniqueName: \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.362851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362685 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2mx\" (UniqueName: \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.362851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362750 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.363001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362844 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.363001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362900 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtrk\" (UniqueName: \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.363001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.362953 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbm9j\" (UniqueName: \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.363126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363088 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.363198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363151 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.363310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363293 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cspn\" (UniqueName: \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.363385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363325 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.363440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363390 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmq8\" (UniqueName: \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.363512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363448 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.363563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.363536 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.382211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.382164 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:00.426776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.426706 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vmj\" (UniqueName: \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\") pod \"maas-api-key-cleanup-29894355-w4d6c\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.464887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.464856 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.465049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.464912 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtrk\" (UniqueName: \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.465049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.464939 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbm9j\" (UniqueName: \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.465049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.464982 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.465049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465016 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.465453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465075 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cspn\" (UniqueName: \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.465453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465092 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.465453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465127 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmq8\" (UniqueName: \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.465453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465219 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.465453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465325 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2mx\" (UniqueName: \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.465782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465719 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.465863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465835 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.465863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465836 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.466008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465966 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.466065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.465997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.498221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.498192 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:00.530123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.530092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cspn\" (UniqueName: \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.530329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.530217 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbm9j\" (UniqueName: \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\") pod \"maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.530412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.530386 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtrk\" (UniqueName: \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.530829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.530804 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmq8\" (UniqueName: \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\") pod \"maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.531293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.531261 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2mx\" (UniqueName: \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.548151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.548105 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:00.556453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.556431 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:00.564381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.562757 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:00.806800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.806762 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:00.815762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.815732 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:00.924550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.924522 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:00.968060 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:00.967609 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a2951b_332d_46d8_8db3_f0e97836fc8d.slice/crio-55840e68e6d870caccada31b9112358f2c007003adcdcecd122022de7e174c33 WatchSource:0}: Error finding container 55840e68e6d870caccada31b9112358f2c007003adcdcecd122022de7e174c33: Status 404 returned error can't find the container with id 55840e68e6d870caccada31b9112358f2c007003adcdcecd122022de7e174c33 Nov 02 23:15:00.968144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:00.968054 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:01.009411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.009381 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:01.011703 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:01.011681 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda780a606_b3e2_4716_9fbd_a98d89cd9416.slice/crio-9343d7988dbe597eccb05d0a8ab81a83d743dcfbbae15134f596cc43ee1d25a0 WatchSource:0}: Error finding container 9343d7988dbe597eccb05d0a8ab81a83d743dcfbbae15134f596cc43ee1d25a0: Status 404 returned error can't find the container with id 9343d7988dbe597eccb05d0a8ab81a83d743dcfbbae15134f596cc43ee1d25a0 Nov 02 23:15:01.014704 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:01.014681 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c9672a_c49c_4506_8d03_c741bed887b6.slice/crio-4c1a142e4e01eccf45105333142ae7a370232470540b65c0560f4d96d52a41b9 WatchSource:0}: Error finding container 4c1a142e4e01eccf45105333142ae7a370232470540b65c0560f4d96d52a41b9: Status 404 returned error can't find the container with id 4c1a142e4e01eccf45105333142ae7a370232470540b65c0560f4d96d52a41b9 Nov 02 23:15:01.018383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.018364 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:01.053064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.053031 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd"] Nov 02 23:15:01.057672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.057644 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ff051d-z8pwd"] Nov 02 23:15:01.188594 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:01.188564 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3566791a_0d40_45f9_8f28_7319e88735db.slice/crio-afe5a31e5be79ff03c0ec05b12f49aa118f0a7776843dfb5176fbc5a075327ce WatchSource:0}: Error finding container afe5a31e5be79ff03c0ec05b12f49aa118f0a7776843dfb5176fbc5a075327ce: Status 404 returned error can't find the container with id afe5a31e5be79ff03c0ec05b12f49aa118f0a7776843dfb5176fbc5a075327ce Nov 02 23:15:01.189536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.189511 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:01.210414 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:01.210389 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917d95ef_ccde_4f71_b49d_4527ff30b9c1.slice/crio-707b8c4a26a982295ba762f43b0dbecd21ece4e707b5fa27014c3b3cf376d921 WatchSource:0}: Error finding container 707b8c4a26a982295ba762f43b0dbecd21ece4e707b5fa27014c3b3cf376d921: Status 404 returned error can't find the container with id 707b8c4a26a982295ba762f43b0dbecd21ece4e707b5fa27014c3b3cf376d921 Nov 02 23:15:01.212095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.212068 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:01.221499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.221473 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerStarted","Data":"707b8c4a26a982295ba762f43b0dbecd21ece4e707b5fa27014c3b3cf376d921"} Nov 02 23:15:01.222949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.222921 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerStarted","Data":"afe5a31e5be79ff03c0ec05b12f49aa118f0a7776843dfb5176fbc5a075327ce"} Nov 02 23:15:01.224410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.224385 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerStarted","Data":"a1de96d25c90d824fadbfbb012a90832f1dd1f9304c633f46e26c89d77b40a40"} Nov 02 23:15:01.225713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.225692 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerStarted","Data":"55840e68e6d870caccada31b9112358f2c007003adcdcecd122022de7e174c33"} Nov 02 23:15:01.226985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.226961 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerStarted","Data":"9343d7988dbe597eccb05d0a8ab81a83d743dcfbbae15134f596cc43ee1d25a0"} Nov 02 23:15:01.228371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.228347 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerStarted","Data":"4c1a142e4e01eccf45105333142ae7a370232470540b65c0560f4d96d52a41b9"} Nov 02 23:15:01.696912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:01.696823 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb0b90b-3d76-499b-991c-5762f35b0945" path="/var/lib/kubelet/pods/8eb0b90b-3d76-499b-991c-5762f35b0945/volumes" Nov 02 23:15:02.062253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.062128 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz"] Nov 02 23:15:02.066678 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.066652 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ce091a-q59jz"] Nov 02 23:15:02.243638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.243518 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerStarted","Data":"3f4f45ba979e3486e6cade350eda0ea861eb236bcc54d4cdb542eb9663ade70c"} Nov 02 23:15:02.249021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.248987 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerStarted","Data":"986f984a4fbe44c133e4848980cbdecfc12a69d82c7930e5b6177f8058f03077"} Nov 02 23:15:02.249313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.249294 4494 scope.go:122] "RemoveContainer" containerID="986f984a4fbe44c133e4848980cbdecfc12a69d82c7930e5b6177f8058f03077" Nov 02 23:15:02.251579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.251547 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerStarted","Data":"a66f66ee4a6af2beeba07162afaa2127906be53a8284fc64d0a3a62f772dcbdf"} Nov 02 23:15:02.251868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.251850 4494 scope.go:122] "RemoveContainer" containerID="a66f66ee4a6af2beeba07162afaa2127906be53a8284fc64d0a3a62f772dcbdf" Nov 02 23:15:02.254903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.254879 4494 generic.go:356] "Generic (PLEG): container finished" podID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerID="a3ca069f5210130aab81182f64c30fd35dafad3d5ab36832ae706e5db3df4593" exitCode=60 Nov 02 23:15:02.255010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.254967 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerDied","Data":"a3ca069f5210130aab81182f64c30fd35dafad3d5ab36832ae706e5db3df4593"} Nov 02 23:15:02.255265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.255233 4494 scope.go:122] "RemoveContainer" containerID="a3ca069f5210130aab81182f64c30fd35dafad3d5ab36832ae706e5db3df4593" Nov 02 23:15:02.261236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.259613 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerStarted","Data":"0cf063adcd828c8fe1223f88bec1c9d300b48df9b0f5ce6d15d1105c60e812b6"} Nov 02 23:15:02.261236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.259971 4494 scope.go:122] "RemoveContainer" containerID="0cf063adcd828c8fe1223f88bec1c9d300b48df9b0f5ce6d15d1105c60e812b6" Nov 02 23:15:02.264129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.264096 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerStarted","Data":"2110819d97bb49ddb81bf1c8a348ab7f6c3774c010ae42f80b5f320cf7f006e8"} Nov 02 23:15:02.367203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.367142 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" podStartSLOduration=2.367128432 podStartE2EDuration="2.367128432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:02.362259909 +0000 UTC m=+1616.792129771" watchObservedRunningTime="2026-11-02 23:15:02.367128432 +0000 UTC m=+1616.796998292" Nov 02 23:15:02.605024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.604991 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4"] Nov 02 23:15:02.609764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.609742 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.664646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.664358 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:15:02.666676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.666649 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4"] Nov 02 23:15:02.703311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.703273 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.703511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.703474 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd54j\" (UniqueName: \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.703708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.703685 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.805482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.805446 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.805685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.805559 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.805761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.805703 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd54j\" (UniqueName: \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.806229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.806202 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.808590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.808564 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.840058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.839949 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" podStartSLOduration=2.839931664 podStartE2EDuration="2.839931664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:02.835814843 +0000 UTC m=+1617.265684704" watchObservedRunningTime="2026-11-02 23:15:02.839931664 +0000 UTC m=+1617.269801534" Nov 02 23:15:02.868404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.868375 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd54j\" (UniqueName: \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\") pod \"maas-api-revoke-sub-3281789d-nzpz4\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:02.924432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:02.924396 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:03.276227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.276190 4494 generic.go:356] "Generic (PLEG): container finished" podID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerID="3f4f45ba979e3486e6cade350eda0ea861eb236bcc54d4cdb542eb9663ade70c" exitCode=60 Nov 02 23:15:03.276732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.276293 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerDied","Data":"3f4f45ba979e3486e6cade350eda0ea861eb236bcc54d4cdb542eb9663ade70c"} Nov 02 23:15:03.276732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.276643 4494 scope.go:122] "RemoveContainer" containerID="3f4f45ba979e3486e6cade350eda0ea861eb236bcc54d4cdb542eb9663ade70c" Nov 02 23:15:03.278553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.278533 4494 generic.go:356] "Generic (PLEG): container finished" podID="3566791a-0d40-45f9-8f28-7319e88735db" containerID="986f984a4fbe44c133e4848980cbdecfc12a69d82c7930e5b6177f8058f03077" exitCode=60 Nov 02 23:15:03.278657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.278616 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerDied","Data":"986f984a4fbe44c133e4848980cbdecfc12a69d82c7930e5b6177f8058f03077"} Nov 02 23:15:03.278718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.278668 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerStarted","Data":"cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0"} Nov 02 23:15:03.281207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.281147 4494 generic.go:356] "Generic (PLEG): container finished" podID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerID="a66f66ee4a6af2beeba07162afaa2127906be53a8284fc64d0a3a62f772dcbdf" exitCode=60 Nov 02 23:15:03.281271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.281197 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerDied","Data":"a66f66ee4a6af2beeba07162afaa2127906be53a8284fc64d0a3a62f772dcbdf"} Nov 02 23:15:03.281271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.281244 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerStarted","Data":"269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a"} Nov 02 23:15:03.283467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.283444 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerStarted","Data":"141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32"} Nov 02 23:15:03.285435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.285412 4494 generic.go:356] "Generic (PLEG): container finished" podID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerID="0cf063adcd828c8fe1223f88bec1c9d300b48df9b0f5ce6d15d1105c60e812b6" exitCode=60 Nov 02 23:15:03.285548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.285502 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerDied","Data":"0cf063adcd828c8fe1223f88bec1c9d300b48df9b0f5ce6d15d1105c60e812b6"} Nov 02 23:15:03.285548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.285541 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerStarted","Data":"ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6"} Nov 02 23:15:03.287309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.287289 4494 generic.go:356] "Generic (PLEG): container finished" podID="61c9672a-c49c-4506-8d03-c741bed887b6" containerID="2110819d97bb49ddb81bf1c8a348ab7f6c3774c010ae42f80b5f320cf7f006e8" exitCode=60 Nov 02 23:15:03.287450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.287429 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerDied","Data":"2110819d97bb49ddb81bf1c8a348ab7f6c3774c010ae42f80b5f320cf7f006e8"} Nov 02 23:15:03.287594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.287580 4494 scope.go:122] "RemoveContainer" containerID="2110819d97bb49ddb81bf1c8a348ab7f6c3774c010ae42f80b5f320cf7f006e8" Nov 02 23:15:03.316419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.316393 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4"] Nov 02 23:15:03.318615 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:15:03.318591 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a9a51a_6a30_468f_b677_876097bed8aa.slice/crio-36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d WatchSource:0}: Error finding container 36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d: Status 404 returned error can't find the container with id 36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d Nov 02 23:15:03.344548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.344522 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:03.519836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.519786 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" podStartSLOduration=3.519773367 podStartE2EDuration="3.519773367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.519258361 +0000 UTC m=+1617.949128229" watchObservedRunningTime="2026-11-02 23:15:03.519773367 +0000 UTC m=+1617.949643227" Nov 02 23:15:03.698936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.698886 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4048069e-453c-4967-b4b9-0e688b705f6b" path="/var/lib/kubelet/pods/4048069e-453c-4967-b4b9-0e688b705f6b/volumes" Nov 02 23:15:03.756376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.756333 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" podStartSLOduration=3.756320086 podStartE2EDuration="3.756320086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.753653032 +0000 UTC m=+1618.183522909" watchObservedRunningTime="2026-11-02 23:15:03.756320086 +0000 UTC m=+1618.186189947" Nov 02 23:15:03.872542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.872480 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" podStartSLOduration=3.872465821 podStartE2EDuration="3.872465821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.870937913 +0000 UTC m=+1618.300807771" watchObservedRunningTime="2026-11-02 23:15:03.872465821 +0000 UTC m=+1618.302335705" Nov 02 23:15:03.988081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:03.988016 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" podStartSLOduration=3.987999924 podStartE2EDuration="3.987999924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.986384972 +0000 UTC m=+1618.416254840" watchObservedRunningTime="2026-11-02 23:15:03.987999924 +0000 UTC m=+1618.417869785" Nov 02 23:15:04.299091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.299050 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerStarted","Data":"7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2"} Nov 02 23:15:04.301099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.301076 4494 generic.go:356] "Generic (PLEG): container finished" podID="3566791a-0d40-45f9-8f28-7319e88735db" containerID="cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0" exitCode=60 Nov 02 23:15:04.301206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.301158 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerDied","Data":"cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0"} Nov 02 23:15:04.301276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.301236 4494 scope.go:122] "RemoveContainer" containerID="986f984a4fbe44c133e4848980cbdecfc12a69d82c7930e5b6177f8058f03077" Nov 02 23:15:04.301476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.301461 4494 scope.go:122] "RemoveContainer" containerID="cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0" Nov 02 23:15:04.301846 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:04.301821 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc_odh-ai-gateway-infra(3566791a-0d40-45f9-8f28-7319e88735db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" podUID="3566791a-0d40-45f9-8f28-7319e88735db" Nov 02 23:15:04.303081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.303056 4494 generic.go:356] "Generic (PLEG): container finished" podID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerID="269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a" exitCode=60 Nov 02 23:15:04.303203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.303151 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerDied","Data":"269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a"} Nov 02 23:15:04.303390 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.303377 4494 scope.go:122] "RemoveContainer" containerID="269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a" Nov 02 23:15:04.303662 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:04.303644 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-w4d6c_odh-ai-gateway-infra(b6ea8701-ac08-45de-b25f-f2e8dce32514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" Nov 02 23:15:04.305161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.305135 4494 generic.go:356] "Generic (PLEG): container finished" podID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerID="141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32" exitCode=60 Nov 02 23:15:04.305296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.305208 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerDied","Data":"141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32"} Nov 02 23:15:04.305415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.305401 4494 scope.go:122] "RemoveContainer" containerID="141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32" Nov 02 23:15:04.305680 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:04.305656 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh_odh-ai-gateway-infra(28a2951b-332d-46d8-8db3-f0e97836fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" Nov 02 23:15:04.307109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.307085 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" event={"ID":"e9a9a51a-6a30-468f-b677-876097bed8aa","Type":"ContainerStarted","Data":"2f2e8f2cbc143cd732836c8a01241ab0d0d2396283c9164feb4d193268f934f1"} Nov 02 23:15:04.307243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.307125 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" event={"ID":"e9a9a51a-6a30-468f-b677-876097bed8aa","Type":"ContainerStarted","Data":"36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d"} Nov 02 23:15:04.309133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.309111 4494 generic.go:356] "Generic (PLEG): container finished" podID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerID="ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6" exitCode=60 Nov 02 23:15:04.309243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.309205 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerDied","Data":"ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6"} Nov 02 23:15:04.309497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.309475 4494 scope.go:122] "RemoveContainer" containerID="ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6" Nov 02 23:15:04.309782 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:04.309761 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx_odh-ai-gateway-infra(a780a606-b3e2-4716-9fbd-a98d89cd9416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" Nov 02 23:15:04.311075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.311051 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerStarted","Data":"1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2"} Nov 02 23:15:04.316450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.316433 4494 scope.go:122] "RemoveContainer" containerID="a66f66ee4a6af2beeba07162afaa2127906be53a8284fc64d0a3a62f772dcbdf" Nov 02 23:15:04.329799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.329786 4494 scope.go:122] "RemoveContainer" containerID="a3ca069f5210130aab81182f64c30fd35dafad3d5ab36832ae706e5db3df4593" Nov 02 23:15:04.482290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.482261 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:04.513255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.513233 4494 scope.go:122] "RemoveContainer" containerID="0cf063adcd828c8fe1223f88bec1c9d300b48df9b0f5ce6d15d1105c60e812b6" Nov 02 23:15:04.887279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:04.887230 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" podStartSLOduration=2.8872150039999998 podStartE2EDuration="2.887215004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:04.883908187 +0000 UTC m=+1619.313778050" watchObservedRunningTime="2026-11-02 23:15:04.887215004 +0000 UTC m=+1619.317084869" Nov 02 23:15:05.051202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.051139 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6"] Nov 02 23:15:05.055610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.055583 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe483e7-nq4p6"] Nov 02 23:15:05.326375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.326345 4494 scope.go:122] "RemoveContainer" containerID="ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6" Nov 02 23:15:05.326804 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.326705 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx_odh-ai-gateway-infra(a780a606-b3e2-4716-9fbd-a98d89cd9416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" Nov 02 23:15:05.328128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.328101 4494 generic.go:356] "Generic (PLEG): container finished" podID="61c9672a-c49c-4506-8d03-c741bed887b6" containerID="1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2" exitCode=60 Nov 02 23:15:05.328263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.328204 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerDied","Data":"1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2"} Nov 02 23:15:05.328318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.328304 4494 scope.go:122] "RemoveContainer" containerID="2110819d97bb49ddb81bf1c8a348ab7f6c3774c010ae42f80b5f320cf7f006e8" Nov 02 23:15:05.328480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.328464 4494 scope.go:122] "RemoveContainer" containerID="1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2" Nov 02 23:15:05.328806 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.328784 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd_odh-ai-gateway-infra(61c9672a-c49c-4506-8d03-c741bed887b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" Nov 02 23:15:05.330304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.330283 4494 generic.go:356] "Generic (PLEG): container finished" podID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerID="7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2" exitCode=60 Nov 02 23:15:05.330395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.330366 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerDied","Data":"7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2"} Nov 02 23:15:05.330562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.330549 4494 scope.go:122] "RemoveContainer" containerID="7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2" Nov 02 23:15:05.330862 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.330842 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n_odh-ai-gateway-infra(917d95ef-ccde-4f71-b49d-4527ff30b9c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" Nov 02 23:15:05.332211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.332193 4494 scope.go:122] "RemoveContainer" containerID="cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0" Nov 02 23:15:05.332502 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.332483 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc_odh-ai-gateway-infra(3566791a-0d40-45f9-8f28-7319e88735db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" podUID="3566791a-0d40-45f9-8f28-7319e88735db" Nov 02 23:15:05.333728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.333708 4494 scope.go:122] "RemoveContainer" containerID="269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a" Nov 02 23:15:05.334001 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.333982 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-w4d6c_odh-ai-gateway-infra(b6ea8701-ac08-45de-b25f-f2e8dce32514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" Nov 02 23:15:05.335363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.335345 4494 scope.go:122] "RemoveContainer" containerID="141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32" Nov 02 23:15:05.335588 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:05.335567 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh_odh-ai-gateway-infra(28a2951b-332d-46d8-8db3-f0e97836fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" Nov 02 23:15:05.336817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.336797 4494 generic.go:356] "Generic (PLEG): container finished" podID="e9a9a51a-6a30-468f-b677-876097bed8aa" containerID="2f2e8f2cbc143cd732836c8a01241ab0d0d2396283c9164feb4d193268f934f1" exitCode=0 Nov 02 23:15:05.336905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.336828 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" event={"ID":"e9a9a51a-6a30-468f-b677-876097bed8aa","Type":"ContainerDied","Data":"2f2e8f2cbc143cd732836c8a01241ab0d0d2396283c9164feb4d193268f934f1"} Nov 02 23:15:05.344479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.344463 4494 scope.go:122] "RemoveContainer" containerID="3f4f45ba979e3486e6cade350eda0ea861eb236bcc54d4cdb542eb9663ade70c" Nov 02 23:15:05.695565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:05.695534 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead488c4-a1f7-4a49-83b9-19c9009d273e" path="/var/lib/kubelet/pods/ead488c4-a1f7-4a49-83b9-19c9009d273e/volumes" Nov 02 23:15:06.347463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.347434 4494 scope.go:122] "RemoveContainer" containerID="7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2" Nov 02 23:15:06.347900 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:06.347819 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n_odh-ai-gateway-infra(917d95ef-ccde-4f71-b49d-4527ff30b9c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" Nov 02 23:15:06.348961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.348937 4494 scope.go:122] "RemoveContainer" containerID="1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2" Nov 02 23:15:06.349405 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:06.349275 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd_odh-ai-gateway-infra(61c9672a-c49c-4506-8d03-c741bed887b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" Nov 02 23:15:06.517017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.516990 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:06.565399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.565370 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\") pod \"e9a9a51a-6a30-468f-b677-876097bed8aa\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " Nov 02 23:15:06.565579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.565542 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\" (UniqueName: \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\") pod \"e9a9a51a-6a30-468f-b677-876097bed8aa\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " Nov 02 23:15:06.565644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.565612 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\") pod \"e9a9a51a-6a30-468f-b677-876097bed8aa\" (UID: \"e9a9a51a-6a30-468f-b677-876097bed8aa\") " Nov 02 23:15:06.566075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.566049 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca" pod "e9a9a51a-6a30-468f-b677-876097bed8aa" (UID: "e9a9a51a-6a30-468f-b677-876097bed8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:06.566263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.566246 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:06.568617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.568591 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca" pod "e9a9a51a-6a30-468f-b677-876097bed8aa" (UID: "e9a9a51a-6a30-468f-b677-876097bed8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:15:06.568617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.568604 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j" pod "e9a9a51a-6a30-468f-b677-876097bed8aa" (UID: "e9a9a51a-6a30-468f-b677-876097bed8aa"). InnerVolumeSpecName "kube-api-access-qd54j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:06.667885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.667858 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd54j\" (UniqueName: \"kubernetes.io/projected/e9a9a51a-6a30-468f-b677-876097bed8aa-kube-api-access-qd54j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:06.667885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.667886 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a9a51a-6a30-468f-b677-876097bed8aa-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:06.810142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:06.810111 4494 scope.go:122] "RemoveContainer" containerID="4ea10cc1126b31804ce9b7c416357b8f28ab1a3ce5e20919de014bffe3a4908b" Nov 02 23:15:07.358189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:07.358142 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" event={"ID":"e9a9a51a-6a30-468f-b677-876097bed8aa","Type":"ContainerDied","Data":"36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d"} Nov 02 23:15:07.358572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:07.358227 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ef548169312a086b6c4a644b1e381c012f1d280f142429c8285beb255e5e8d" Nov 02 23:15:07.358572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:07.358159 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4" Nov 02 23:15:08.258110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:08.258079 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:13.173770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:13.173738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:14.687109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:14.687080 4494 scope.go:122] "RemoveContainer" containerID="269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a" Nov 02 23:15:14.687650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:14.687456 4494 scope.go:122] "RemoveContainer" containerID="141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32" Nov 02 23:15:14.687650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:14.687547 4494 scope.go:122] "RemoveContainer" containerID="ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6" Nov 02 23:15:15.441230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:15.441184 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerStarted","Data":"2fc7691a5bf1ef69eb57a7809cf11b6114962f8b2c86027002fe53905df8d2df"} Nov 02 23:15:15.443239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:15.443210 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerStarted","Data":"1053e66404b15a450dc291c6901478d6224fa7a596c6b092097dc083ae6cacd5"} Nov 02 23:15:15.445321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:15.445301 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerStarted","Data":"fa1724b4a12c5c8f4c69323fec3cbd48fd1d7a21cc4383096d3f4731cd9012df"} Nov 02 23:15:16.049314 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.049281 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk"] Nov 02 23:15:16.053506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.053454 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77bc78f0-k9qkk"] Nov 02 23:15:16.455107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.455074 4494 generic.go:356] "Generic (PLEG): container finished" podID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerID="2fc7691a5bf1ef69eb57a7809cf11b6114962f8b2c86027002fe53905df8d2df" exitCode=60 Nov 02 23:15:16.455306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.455151 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerDied","Data":"2fc7691a5bf1ef69eb57a7809cf11b6114962f8b2c86027002fe53905df8d2df"} Nov 02 23:15:16.455306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.455250 4494 scope.go:122] "RemoveContainer" containerID="269633b3fc8dc7d022374ffe3b97d5dfab68e8cdc711d9f9ce41c9529802562a" Nov 02 23:15:16.455644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.455622 4494 scope.go:122] "RemoveContainer" containerID="2fc7691a5bf1ef69eb57a7809cf11b6114962f8b2c86027002fe53905df8d2df" Nov 02 23:15:16.456042 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:16.456019 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-w4d6c_odh-ai-gateway-infra(b6ea8701-ac08-45de-b25f-f2e8dce32514)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" Nov 02 23:15:16.457359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.457339 4494 generic.go:356] "Generic (PLEG): container finished" podID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerID="1053e66404b15a450dc291c6901478d6224fa7a596c6b092097dc083ae6cacd5" exitCode=60 Nov 02 23:15:16.457458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.457415 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerDied","Data":"1053e66404b15a450dc291c6901478d6224fa7a596c6b092097dc083ae6cacd5"} Nov 02 23:15:16.457745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.457725 4494 scope.go:122] "RemoveContainer" containerID="1053e66404b15a450dc291c6901478d6224fa7a596c6b092097dc083ae6cacd5" Nov 02 23:15:16.458081 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:16.458064 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh_odh-ai-gateway-infra(28a2951b-332d-46d8-8db3-f0e97836fc8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" Nov 02 23:15:16.459303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.459281 4494 generic.go:356] "Generic (PLEG): container finished" podID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerID="fa1724b4a12c5c8f4c69323fec3cbd48fd1d7a21cc4383096d3f4731cd9012df" exitCode=60 Nov 02 23:15:16.459411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.459351 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerDied","Data":"fa1724b4a12c5c8f4c69323fec3cbd48fd1d7a21cc4383096d3f4731cd9012df"} Nov 02 23:15:16.459650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.459637 4494 scope.go:122] "RemoveContainer" containerID="fa1724b4a12c5c8f4c69323fec3cbd48fd1d7a21cc4383096d3f4731cd9012df" Nov 02 23:15:16.459889 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:16.459874 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx_odh-ai-gateway-infra(a780a606-b3e2-4716-9fbd-a98d89cd9416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" Nov 02 23:15:16.469302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.469284 4494 scope.go:122] "RemoveContainer" containerID="141779eec848006c22dd95505e1011743a4c0e356066a2ca943a620e286bbe32" Nov 02 23:15:16.482645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.482624 4494 scope.go:122] "RemoveContainer" containerID="ee5ae27cfc9f992f3ca7f7adebe9eff6fb48961e5e843039e8741ff75094f6c6" Nov 02 23:15:16.570619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.570585 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:16.685683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.685650 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:16.686249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.686233 4494 scope.go:122] "RemoveContainer" containerID="cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0" Nov 02 23:15:16.686388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.686373 4494 scope.go:122] "RemoveContainer" containerID="7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2" Nov 02 23:15:16.801447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:16.801421 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:17.060302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.060253 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds"] Nov 02 23:15:17.061430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.061407 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2975ef91-5tlds"] Nov 02 23:15:17.672495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.670703 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerStarted","Data":"ede54c39ead4b4fac0c89d0b1f0c4cbdc0c2d1f01ca20d56114a74e4ca09cf02"} Nov 02 23:15:17.680196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.676892 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerStarted","Data":"f5addefbdabf336c4330768ac841d8a2aa05f98abcdcb3c90f06d1702ba0275c"} Nov 02 23:15:17.706997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.703519 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be" path="/var/lib/kubelet/pods/9e4f0a6f-4e46-4cdc-bf52-a70e9e1877be/volumes" Nov 02 23:15:17.708380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:17.708342 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7292e94-19f0-4336-a312-39ee18b66d5a" path="/var/lib/kubelet/pods/c7292e94-19f0-4336-a312-39ee18b66d5a/volumes" Nov 02 23:15:17.732577 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:17.732537 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:15:18.069037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.069010 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b"] Nov 02 23:15:18.072440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.072401 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:18.073712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.073687 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee1a93d-nkg8b"] Nov 02 23:15:18.164142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.164111 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:18.166857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.166836 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:18.182700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.182674 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\") pod \"28a2951b-332d-46d8-8db3-f0e97836fc8d\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " Nov 02 23:15:18.182806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.182733 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\" (UniqueName: \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\") pod \"b6ea8701-ac08-45de-b25f-f2e8dce32514\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " Nov 02 23:15:18.182806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.182766 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\") pod \"b6ea8701-ac08-45de-b25f-f2e8dce32514\" (UID: \"b6ea8701-ac08-45de-b25f-f2e8dce32514\") " Nov 02 23:15:18.182923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.182814 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\" (UniqueName: \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\") pod \"28a2951b-332d-46d8-8db3-f0e97836fc8d\" (UID: \"28a2951b-332d-46d8-8db3-f0e97836fc8d\") " Nov 02 23:15:18.183069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.183041 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca" pod "28a2951b-332d-46d8-8db3-f0e97836fc8d" (UID: "28a2951b-332d-46d8-8db3-f0e97836fc8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.183200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.183158 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca" pod "b6ea8701-ac08-45de-b25f-f2e8dce32514" (UID: "b6ea8701-ac08-45de-b25f-f2e8dce32514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.183384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.183366 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a2951b-332d-46d8-8db3-f0e97836fc8d-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.183440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.183389 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ea8701-ac08-45de-b25f-f2e8dce32514-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.184876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.184857 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj" pod "b6ea8701-ac08-45de-b25f-f2e8dce32514" (UID: "b6ea8701-ac08-45de-b25f-f2e8dce32514"). InnerVolumeSpecName "kube-api-access-s9vmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.185014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.184998 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx" pod "28a2951b-332d-46d8-8db3-f0e97836fc8d" (UID: "28a2951b-332d-46d8-8db3-f0e97836fc8d"). InnerVolumeSpecName "kube-api-access-nq2mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.284424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284340 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\" (UniqueName: \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\") pod \"a780a606-b3e2-4716-9fbd-a98d89cd9416\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " Nov 02 23:15:18.284590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284513 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\") pod \"a780a606-b3e2-4716-9fbd-a98d89cd9416\" (UID: \"a780a606-b3e2-4716-9fbd-a98d89cd9416\") " Nov 02 23:15:18.284836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284812 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca" pod "a780a606-b3e2-4716-9fbd-a98d89cd9416" (UID: "a780a606-b3e2-4716-9fbd-a98d89cd9416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.284980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284965 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9vmj\" (UniqueName: \"kubernetes.io/projected/b6ea8701-ac08-45de-b25f-f2e8dce32514-kube-api-access-s9vmj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.285262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284981 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2mx\" (UniqueName: \"kubernetes.io/projected/28a2951b-332d-46d8-8db3-f0e97836fc8d-kube-api-access-nq2mx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.285262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.284991 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a780a606-b3e2-4716-9fbd-a98d89cd9416-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.286886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.286861 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8" pod "a780a606-b3e2-4716-9fbd-a98d89cd9416" (UID: "a780a606-b3e2-4716-9fbd-a98d89cd9416"). InnerVolumeSpecName "kube-api-access-5jmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.387060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.387025 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmq8\" (UniqueName: \"kubernetes.io/projected/a780a606-b3e2-4716-9fbd-a98d89cd9416-kube-api-access-5jmq8\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.456392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.456357 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:18.687060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.687028 4494 scope.go:122] "RemoveContainer" containerID="1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2" Nov 02 23:15:18.699028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.699004 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" event={"ID":"28a2951b-332d-46d8-8db3-f0e97836fc8d","Type":"ContainerDied","Data":"55840e68e6d870caccada31b9112358f2c007003adcdcecd122022de7e174c33"} Nov 02 23:15:18.699145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.699069 4494 scope.go:122] "RemoveContainer" containerID="1053e66404b15a450dc291c6901478d6224fa7a596c6b092097dc083ae6cacd5" Nov 02 23:15:18.699145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.699008 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh" Nov 02 23:15:18.700819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.700799 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" event={"ID":"a780a606-b3e2-4716-9fbd-a98d89cd9416","Type":"ContainerDied","Data":"9343d7988dbe597eccb05d0a8ab81a83d743dcfbbae15134f596cc43ee1d25a0"} Nov 02 23:15:18.700933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.700882 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx" Nov 02 23:15:18.707307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.707283 4494 generic.go:356] "Generic (PLEG): container finished" podID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerID="ede54c39ead4b4fac0c89d0b1f0c4cbdc0c2d1f01ca20d56114a74e4ca09cf02" exitCode=60 Nov 02 23:15:18.707440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.707352 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerDied","Data":"ede54c39ead4b4fac0c89d0b1f0c4cbdc0c2d1f01ca20d56114a74e4ca09cf02"} Nov 02 23:15:18.707774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.707759 4494 scope.go:122] "RemoveContainer" containerID="ede54c39ead4b4fac0c89d0b1f0c4cbdc0c2d1f01ca20d56114a74e4ca09cf02" Nov 02 23:15:18.708070 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:18.708049 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n_odh-ai-gateway-infra(917d95ef-ccde-4f71-b49d-4527ff30b9c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" Nov 02 23:15:18.709801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.709778 4494 generic.go:356] "Generic (PLEG): container finished" podID="3566791a-0d40-45f9-8f28-7319e88735db" containerID="f5addefbdabf336c4330768ac841d8a2aa05f98abcdcb3c90f06d1702ba0275c" exitCode=60 Nov 02 23:15:18.709895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.709844 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerDied","Data":"f5addefbdabf336c4330768ac841d8a2aa05f98abcdcb3c90f06d1702ba0275c"} Nov 02 23:15:18.710271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.710252 4494 scope.go:122] "RemoveContainer" containerID="f5addefbdabf336c4330768ac841d8a2aa05f98abcdcb3c90f06d1702ba0275c" Nov 02 23:15:18.710545 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:18.710521 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc_odh-ai-gateway-infra(3566791a-0d40-45f9-8f28-7319e88735db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" podUID="3566791a-0d40-45f9-8f28-7319e88735db" Nov 02 23:15:18.711445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.711422 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" event={"ID":"b6ea8701-ac08-45de-b25f-f2e8dce32514","Type":"ContainerDied","Data":"a1de96d25c90d824fadbfbb012a90832f1dd1f9304c633f46e26c89d77b40a40"} Nov 02 23:15:18.711445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.711444 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c" Nov 02 23:15:18.713302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.713280 4494 scope.go:122] "RemoveContainer" containerID="fa1724b4a12c5c8f4c69323fec3cbd48fd1d7a21cc4383096d3f4731cd9012df" Nov 02 23:15:18.724639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.724620 4494 scope.go:122] "RemoveContainer" containerID="7d8fa37e525a985d0a5d0e7b2a4811fb934b0b1cc87539a0a4aef92fdad7dcb2" Nov 02 23:15:18.786534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.786514 4494 scope.go:122] "RemoveContainer" containerID="cf070305ab2c49b26e35b33684c8ae7d9340115c25160690e69a9f960435ffe0" Nov 02 23:15:18.800672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.800651 4494 scope.go:122] "RemoveContainer" containerID="2fc7691a5bf1ef69eb57a7809cf11b6114962f8b2c86027002fe53905df8d2df" Nov 02 23:15:18.883191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:18.883142 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:19.006554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.006519 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:19.222818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.222781 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:19.227685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.227626 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894355-4slbh"] Nov 02 23:15:19.357199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.356420 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:19.361766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.361735 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5153fe-29894355-79gdx"] Nov 02 23:15:19.462891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.462856 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:19.467566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.467530 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-w4d6c"] Nov 02 23:15:19.696475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.696437 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" path="/var/lib/kubelet/pods/28a2951b-332d-46d8-8db3-f0e97836fc8d/volumes" Nov 02 23:15:19.697127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.697099 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307c5897-ae78-484c-8ddc-801d53a138fe" path="/var/lib/kubelet/pods/307c5897-ae78-484c-8ddc-801d53a138fe/volumes" Nov 02 23:15:19.697633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.697618 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" path="/var/lib/kubelet/pods/a780a606-b3e2-4716-9fbd-a98d89cd9416/volumes" Nov 02 23:15:19.698495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.698478 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" path="/var/lib/kubelet/pods/b6ea8701-ac08-45de-b25f-f2e8dce32514/volumes" Nov 02 23:15:19.723460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.723433 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerStarted","Data":"82e6433bd64278abfe937c61f6ca7ecab78d748e40ebb0588bdf23b22211e69b"} Nov 02 23:15:19.986607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:19.986574 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:20.067583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.067558 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:20.172811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.172772 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\" (UniqueName: \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\") pod \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " Nov 02 23:15:20.173020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.172848 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\" (UniqueName: \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\") pod \"3566791a-0d40-45f9-8f28-7319e88735db\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " Nov 02 23:15:20.173020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.172881 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\") pod \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\" (UID: \"917d95ef-ccde-4f71-b49d-4527ff30b9c1\") " Nov 02 23:15:20.173020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.173009 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\") pod \"3566791a-0d40-45f9-8f28-7319e88735db\" (UID: \"3566791a-0d40-45f9-8f28-7319e88735db\") " Nov 02 23:15:20.173420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.173390 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca" pod "917d95ef-ccde-4f71-b49d-4527ff30b9c1" (UID: "917d95ef-ccde-4f71-b49d-4527ff30b9c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:20.173495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.173390 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca" pod "3566791a-0d40-45f9-8f28-7319e88735db" (UID: "3566791a-0d40-45f9-8f28-7319e88735db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:20.173750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.173724 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3566791a-0d40-45f9-8f28-7319e88735db-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:20.173750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.173751 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/917d95ef-ccde-4f71-b49d-4527ff30b9c1-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:20.175118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.175096 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk" pod "917d95ef-ccde-4f71-b49d-4527ff30b9c1" (UID: "917d95ef-ccde-4f71-b49d-4527ff30b9c1"). InnerVolumeSpecName "kube-api-access-8jtrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:20.175209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.175190 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn" pod "3566791a-0d40-45f9-8f28-7319e88735db" (UID: "3566791a-0d40-45f9-8f28-7319e88735db"). InnerVolumeSpecName "kube-api-access-9cspn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:20.275425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.275345 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtrk\" (UniqueName: \"kubernetes.io/projected/917d95ef-ccde-4f71-b49d-4527ff30b9c1-kube-api-access-8jtrk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:20.275425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.275373 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cspn\" (UniqueName: \"kubernetes.io/projected/3566791a-0d40-45f9-8f28-7319e88735db-kube-api-access-9cspn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:20.771228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.771189 4494 generic.go:356] "Generic (PLEG): container finished" podID="61c9672a-c49c-4506-8d03-c741bed887b6" containerID="82e6433bd64278abfe937c61f6ca7ecab78d748e40ebb0588bdf23b22211e69b" exitCode=60 Nov 02 23:15:20.771228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.771211 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerDied","Data":"82e6433bd64278abfe937c61f6ca7ecab78d748e40ebb0588bdf23b22211e69b"} Nov 02 23:15:20.771466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.771280 4494 scope.go:122] "RemoveContainer" containerID="1ecd0ebe7ef2452a8f719be135a34a40def627d42bf1bbf5e95c653362fc4de2" Nov 02 23:15:20.771810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.771779 4494 scope.go:122] "RemoveContainer" containerID="82e6433bd64278abfe937c61f6ca7ecab78d748e40ebb0588bdf23b22211e69b" Nov 02 23:15:20.772271 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:20.772232 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd_odh-ai-gateway-infra(61c9672a-c49c-4506-8d03-c741bed887b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" Nov 02 23:15:20.773088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.773068 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" event={"ID":"917d95ef-ccde-4f71-b49d-4527ff30b9c1","Type":"ContainerDied","Data":"707b8c4a26a982295ba762f43b0dbecd21ece4e707b5fa27014c3b3cf376d921"} Nov 02 23:15:20.773088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.773079 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n" Nov 02 23:15:20.774660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.774640 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" event={"ID":"3566791a-0d40-45f9-8f28-7319e88735db","Type":"ContainerDied","Data":"afe5a31e5be79ff03c0ec05b12f49aa118f0a7776843dfb5176fbc5a075327ce"} Nov 02 23:15:20.774746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.774679 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc" Nov 02 23:15:20.785691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.785671 4494 scope.go:122] "RemoveContainer" containerID="ede54c39ead4b4fac0c89d0b1f0c4cbdc0c2d1f01ca20d56114a74e4ca09cf02" Nov 02 23:15:20.796587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.796564 4494 scope.go:122] "RemoveContainer" containerID="f5addefbdabf336c4330768ac841d8a2aa05f98abcdcb3c90f06d1702ba0275c" Nov 02 23:15:20.880239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:20.880209 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:21.007069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.007041 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:21.011745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.011721 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894355-8mmdc"] Nov 02 23:15:21.121720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.121642 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:21.125646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.125623 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42983-29894355-j785n"] Nov 02 23:15:21.694932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.694901 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3566791a-0d40-45f9-8f28-7319e88735db" path="/var/lib/kubelet/pods/3566791a-0d40-45f9-8f28-7319e88735db/volumes" Nov 02 23:15:21.695430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.695414 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" path="/var/lib/kubelet/pods/917d95ef-ccde-4f71-b49d-4527ff30b9c1/volumes" Nov 02 23:15:21.722975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:21.722948 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:22.112972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.112942 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:22.306743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.306656 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\") pod \"61c9672a-c49c-4506-8d03-c741bed887b6\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " Nov 02 23:15:22.306935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.306826 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\" (UniqueName: \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\") pod \"61c9672a-c49c-4506-8d03-c741bed887b6\" (UID: \"61c9672a-c49c-4506-8d03-c741bed887b6\") " Nov 02 23:15:22.307040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.307012 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca" pod "61c9672a-c49c-4506-8d03-c741bed887b6" (UID: "61c9672a-c49c-4506-8d03-c741bed887b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:22.307394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.307372 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c9672a-c49c-4506-8d03-c741bed887b6-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:22.309104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.309079 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j" pod "61c9672a-c49c-4506-8d03-c741bed887b6" (UID: "61c9672a-c49c-4506-8d03-c741bed887b6"). InnerVolumeSpecName "kube-api-access-tbm9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:22.408316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.408278 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbm9j\" (UniqueName: \"kubernetes.io/projected/61c9672a-c49c-4506-8d03-c741bed887b6-kube-api-access-tbm9j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:22.814491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.814454 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" Nov 02 23:15:22.814949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.814579 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd" event={"ID":"61c9672a-c49c-4506-8d03-c741bed887b6","Type":"ContainerDied","Data":"4c1a142e4e01eccf45105333142ae7a370232470540b65c0560f4d96d52a41b9"} Nov 02 23:15:22.814949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.814639 4494 scope.go:122] "RemoveContainer" containerID="82e6433bd64278abfe937c61f6ca7ecab78d748e40ebb0588bdf23b22211e69b" Nov 02 23:15:22.985316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.985235 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:22.991525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:22.991499 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-8e34df-29894355-n4qzd"] Nov 02 23:15:23.624301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:23.624267 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:23.697106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:23.697069 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" path="/var/lib/kubelet/pods/61c9672a-c49c-4506-8d03-c741bed887b6/volumes" Nov 02 23:15:23.828887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:23.828793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:23.987346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:23.983883 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:24.136250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:24.133069 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:24.884785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:24.884752 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:15:27.940199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.939343 4494 generic.go:356] "Generic (PLEG): container finished" podID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerID="2b5bb72e6da84bfd9a046bb0c506d404cd42f6dbca23b770245defda35771ae0" exitCode=0 Nov 02 23:15:27.940199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.939684 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerDied","Data":"2b5bb72e6da84bfd9a046bb0c506d404cd42f6dbca23b770245defda35771ae0"} Nov 02 23:15:27.949216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.941834 4494 scope.go:122] "RemoveContainer" containerID="2b5bb72e6da84bfd9a046bb0c506d404cd42f6dbca23b770245defda35771ae0" Nov 02 23:15:27.963201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.956588 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:15:27.963201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.956908 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:15:27.963201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:27.957061 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:15:28.964886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:28.964854 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerStarted","Data":"53403f60e6d92263e0a0f3c35fde94aecd2a275375fe27f968837994f87ccb0e"} Nov 02 23:15:29.073422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.073390 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6"] Nov 02 23:15:29.077251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.077223 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434942fd-5b8l6"] Nov 02 23:15:29.457513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.457467 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:29.617314 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.614298 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:29.671196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.670663 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:15:29.695815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.695786 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bab6e0-cac2-43d1-bae4-aa178d8239bf" path="/var/lib/kubelet/pods/89bab6e0-cac2-43d1-bae4-aa178d8239bf/volumes" Nov 02 23:15:29.845068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.844498 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:15:29.845068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.844613 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:29.845068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.844674 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:29.956831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:29.956787 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:30.143959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:30.143870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:30.326858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:30.326809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:30.853617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:30.853580 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:30.853909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:30.853876 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:31.839208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:31.838609 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:31.839208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:31.838669 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:32.847565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:32.847529 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:32.848218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:32.848158 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:33.837950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:33.837904 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:33.838254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:33.838223 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:34.671799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:34.671758 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" podUID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" containerName="payload-processing" containerID="cri-o://9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045" gracePeriod=25 Nov 02 23:15:34.837834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:34.837797 4494 patch_prober.go:29] interesting pod/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 23:15:34.838021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:34.837865 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 23:15:35.002937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.002906 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:15:35.073587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.073545 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\" (UniqueName: \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\") pod \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " Nov 02 23:15:35.073998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.073972 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\") pod \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\" (UID: \"2a09a9bc-900a-45a1-bf3a-8f75da4d1197\") " Nov 02 23:15:35.074368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.074343 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume" pod "2a09a9bc-900a-45a1-bf3a-8f75da4d1197" (UID: "2a09a9bc-900a-45a1-bf3a-8f75da4d1197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:35.075065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.075037 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:35.075975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.075947 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb" pod "2a09a9bc-900a-45a1-bf3a-8f75da4d1197" (UID: "2a09a9bc-900a-45a1-bf3a-8f75da4d1197"). InnerVolumeSpecName "kube-api-access-72bvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:35.097202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.097160 4494 generic.go:356] "Generic (PLEG): container finished" podID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" containerID="9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045" exitCode=0 Nov 02 23:15:35.097319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.097198 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" event={"ID":"2a09a9bc-900a-45a1-bf3a-8f75da4d1197","Type":"ContainerDied","Data":"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045"} Nov 02 23:15:35.097319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.097243 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" Nov 02 23:15:35.097319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.097290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w" event={"ID":"2a09a9bc-900a-45a1-bf3a-8f75da4d1197","Type":"ContainerDied","Data":"649bcdc59a650cc4aa8d4c853b6feaaf79403b62e905c421caeee2b521c8adc4"} Nov 02 23:15:35.097470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.097325 4494 scope.go:122] "RemoveContainer" containerID="9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045" Nov 02 23:15:35.115238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.115219 4494 scope.go:122] "RemoveContainer" containerID="9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045" Nov 02 23:15:35.115533 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:35.115514 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045\": container with ID starting with 9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045 not found: ID does not exist" containerID="9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045" Nov 02 23:15:35.115593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.115542 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045"} err="failed to get container status \"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045\": rpc error: code = NotFound desc = could not find container \"9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045\": container with ID starting with 9a26f445650766aa645c14a48a6f73065e861970842eb49dccfa8c826077c045 not found: ID does not exist" Nov 02 23:15:35.177146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.177109 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bvb\" (UniqueName: \"kubernetes.io/projected/2a09a9bc-900a-45a1-bf3a-8f75da4d1197-kube-api-access-72bvb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:35.235464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.235365 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:15:35.241032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.241005 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8e34df-85864f97cf-tnf4w"] Nov 02 23:15:35.630780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.630695 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:35.751050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.750580 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" path="/var/lib/kubelet/pods/2a09a9bc-900a-45a1-bf3a-8f75da4d1197/volumes" Nov 02 23:15:35.842595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.842569 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:15:35.843061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.843049 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:15:35.843675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:35.843654 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:15:36.219306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:36.219270 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:37.048453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:37.048426 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq"] Nov 02 23:15:37.098488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:37.098458 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959a96de-dtnsq"] Nov 02 23:15:37.696924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:37.696894 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c63d5fc-6676-4890-8e71-46c2db2028b5" path="/var/lib/kubelet/pods/1c63d5fc-6676-4890-8e71-46c2db2028b5/volumes" Nov 02 23:15:38.046994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:38.046911 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx"] Nov 02 23:15:38.098134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:38.098100 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d1c509-b5bsx"] Nov 02 23:15:39.048752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:39.048720 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw"] Nov 02 23:15:39.099076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:39.099042 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dafbca2-w7dgw"] Nov 02 23:15:39.695526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:39.695496 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302b3db7-b2d9-4faa-878d-d023b6706e44" path="/var/lib/kubelet/pods/302b3db7-b2d9-4faa-878d-d023b6706e44/volumes" Nov 02 23:15:39.695984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:39.695971 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1e57d5-4bc6-4bd0-87e5-a3b866099806" path="/var/lib/kubelet/pods/3e1e57d5-4bc6-4bd0-87e5-a3b866099806/volumes" Nov 02 23:15:40.529897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:40.529867 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:45.442292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:45.442254 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:46.048801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:46.048764 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w"] Nov 02 23:15:46.098924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:46.098893 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71fab6c9-8sm9w"] Nov 02 23:15:47.562598 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:47.562493 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:15:47.694749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:47.694716 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb5afbe-790c-476d-91e1-d52c9cbf01f8" path="/var/lib/kubelet/pods/5cb5afbe-790c-476d-91e1-d52c9cbf01f8/volumes" Nov 02 23:15:50.363199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:50.362407 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:53.905478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:53.905442 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:15:53.906017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:53.905650 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" containerID="cri-o://52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1" gracePeriod=30 Nov 02 23:15:55.466985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:55.466936 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:15:59.197972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.197940 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:15:59.288763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.288724 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.288961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.288842 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.288961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.288872 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.288961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.288919 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289042 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289082 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289130 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289162 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289235 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.289359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289282 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.289359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289326 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:59.289567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289382 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\") pod \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\" (UID: \"c0f44320-2b49-42dd-845e-b4a06cc3a0b4\") " Nov 02 23:15:59.289567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289490 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.289567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.289503 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.290118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.290026 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.290118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.290042 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.290118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.290051 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.290118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.290060 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.290118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.290069 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.292075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.292045 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:15:59.292362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.292331 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:15:59.292618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.292594 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:59.292697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.292677 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w" pod "c0f44320-2b49-42dd-845e-b4a06cc3a0b4" (UID: "c0f44320-2b49-42dd-845e-b4a06cc3a0b4"). InnerVolumeSpecName "kube-api-access-d7j9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.391733 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.391768 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7j9w\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-kube-api-access-d7j9w\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.391784 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.391797 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f44320-2b49-42dd-845e-b4a06cc3a0b4-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.393120 4494 generic.go:356] "Generic (PLEG): container finished" podID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerID="52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1" exitCode=0 Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.393462 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" event={"ID":"c0f44320-2b49-42dd-845e-b4a06cc3a0b4","Type":"ContainerDied","Data":"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1"} Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.393514 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" event={"ID":"c0f44320-2b49-42dd-845e-b4a06cc3a0b4","Type":"ContainerDied","Data":"9da022af49e9c979fc269dfa5fae5fdd7d3b20d00245a59d39a47b2e8717a698"} Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.393546 4494 scope.go:122] "RemoveContainer" containerID="52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1" Nov 02 23:15:59.396203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.393686 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l" Nov 02 23:15:59.418060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.418035 4494 scope.go:122] "RemoveContainer" containerID="52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1" Nov 02 23:15:59.418506 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:15:59.418414 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1\": container with ID starting with 52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1 not found: ID does not exist" containerID="52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1" Nov 02 23:15:59.418506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.418456 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1"} err="failed to get container status \"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1\": rpc error: code = NotFound desc = could not find container \"52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1\": container with ID starting with 52902fb88f82bd33939c7d0440302a42244f2dd324ee3075a660fd3fabd521d1 not found: ID does not exist" Nov 02 23:15:59.569605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.569511 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:15:59.573001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.572975 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-rjl8l"] Nov 02 23:15:59.696803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.696767 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" path="/var/lib/kubelet/pods/c0f44320-2b49-42dd-845e-b4a06cc3a0b4/volumes" Nov 02 23:15:59.779405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:15:59.779368 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:00.991227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:00.991086 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:02.641830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:02.641804 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:02.936432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:02.936396 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:03.083824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.083591 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:03.237313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.237232 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn"] Nov 02 23:16:03.287387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.287352 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7a67875-ssswn"] Nov 02 23:16:03.495587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.495511 4494 generic.go:356] "Generic (PLEG): container finished" podID="3407dc7f-387d-4f50-950b-fa841a010942" containerID="2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3" exitCode=0 Nov 02 23:16:03.495743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.495590 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerDied","Data":"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3"} Nov 02 23:16:03.496074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.496058 4494 scope.go:122] "RemoveContainer" containerID="2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3" Nov 02 23:16:03.497768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.497742 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:03.497838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.497820 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:03.497875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.497857 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:03.697115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:03.697077 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24f22a1-cb47-4c97-aab3-338d60c124f0" path="/var/lib/kubelet/pods/d24f22a1-cb47-4c97-aab3-338d60c124f0/volumes" Nov 02 23:16:04.506254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:04.506215 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerStarted","Data":"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7"} Nov 02 23:16:04.827932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:04.827837 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:04.990257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:04.990218 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:16:04.991477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:04.991448 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:16:04.991612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:04.991504 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 23:16:05.990374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:05.990339 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:16:05.990800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:05.990407 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 23:16:06.858085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.858051 4494 scope.go:122] "RemoveContainer" containerID="6ce3e462c117db536169972444aa8cbf6096ba5dd6bae755029307990f77b72c" Nov 02 23:16:06.869985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.869956 4494 scope.go:122] "RemoveContainer" containerID="3429c0bcfa26e1da50f4858ed57f0b550fc3483741562021a293f94d90139f9f" Nov 02 23:16:06.880022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.880004 4494 scope.go:122] "RemoveContainer" containerID="770650fd40675826443aa8f1dc7d9e6b8112dbe7ee5a75666431837ae6b65c24" Nov 02 23:16:06.889472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.889444 4494 scope.go:122] "RemoveContainer" containerID="bf41f197bd6549dedd7438a7f188916f2b966a323772f25a2dc99a9efc95a1cb" Nov 02 23:16:06.898509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.898485 4494 scope.go:122] "RemoveContainer" containerID="0dbdf261fad0f6501d1ef3edcca656c74667132e843320fdc2d1f874131f2b4b" Nov 02 23:16:06.908533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.908508 4494 scope.go:122] "RemoveContainer" containerID="3c8f8f55a9a7f2e915237bf460b4cef279cedfbb9ce70fb1076f5aa29ef044bc" Nov 02 23:16:06.918129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.918112 4494 scope.go:122] "RemoveContainer" containerID="b2d69ef365d30064fb256191855f417d7f928d3c25811741fdfecdbdf41d995f" Nov 02 23:16:06.918217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.918140 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:16:06.927052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.927035 4494 scope.go:122] "RemoveContainer" containerID="461da9200514ce91463ef28c2c93d1b68d8729bd35522a01bc13bee6c3eb4dba" Nov 02 23:16:06.969446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.969423 4494 scope.go:122] "RemoveContainer" containerID="22b82c6d50a4e1b7a47abae2ffe587233b8abbd72515ec39880b875e49469323" Nov 02 23:16:06.990502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.990476 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:16:06.990814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:06.990529 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 23:16:07.047208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:07.044054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:07.990230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:07.990151 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:16:07.990464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:07.990262 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 23:16:08.269453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:08.269347 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:08.990704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:08.990664 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 23:16:08.990876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:08.990754 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 23:16:09.249273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:09.247336 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:09.393365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:09.393326 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:10.021596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:10.021568 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:16:10.036494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:10.036465 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:16:10.081833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:10.081793 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:16:10.948162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:10.948112 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:11.340291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340218 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m"] Nov 02 23:16:11.340876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340863 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340878 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340887 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e9a9a51a-6a30-468f-b677-876097bed8aa" containerName="revoke-keys" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340891 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a9a51a-6a30-468f-b677-876097bed8aa" containerName="revoke-keys" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340901 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340906 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.340918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340919 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340924 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340932 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340936 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340950 4494 cpu_manager.go:414] "containerMap: removing container" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340954 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340961 4494 cpu_manager.go:414] "containerMap: removing container" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340966 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340978 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340982 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340988 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340991 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.340999 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341003 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341012 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341016 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341022 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341027 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341040 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341044 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341057 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341061 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341086 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341091 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341101 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341106 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341115 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341119 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341132 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" containerName="payload-processing" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341136 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" containerName="payload-processing" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341239 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341248 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341255 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341262 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a09a9bc-900a-45a1-bf3a-8f75da4d1197" containerName="payload-processing" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341269 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a780a606-b3e2-4716-9fbd-a98d89cd9416" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341273 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a9a51a-6a30-468f-b677-876097bed8aa" containerName="revoke-keys" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341278 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341283 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341288 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f44320-2b49-42dd-845e-b4a06cc3a0b4" containerName="istio-proxy" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341294 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341300 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341304 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341309 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341315 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341320 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341325 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9672a-c49c-4506-8d03-c741bed887b6" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341331 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341335 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:11.341848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.341344 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:11.348115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.348090 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.430636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.430603 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:16:11.430912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.430885 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" containerID="cri-o://6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5" gracePeriod=30 Nov 02 23:16:11.437253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.437223 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:16:11.437449 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.437419 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:16:11.441950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.441920 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl7j\" (UniqueName: \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.442110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.442089 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.442229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.442207 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:11.442311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.442263 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.442369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.442335 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m"] Nov 02 23:16:11.579199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.555308 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl7j\" (UniqueName: \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.633166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.633069 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.634566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.634535 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.636901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.634293 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.637088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.625234 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:11.688662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.688625 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.700299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.700266 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl7j\" (UniqueName: \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\") pod \"maas-api-revoke-sub-847ba4df-4d48m\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:11.907691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.907600 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:11.962133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:11.962105 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:12.138020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.137988 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:16:12.138232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.138212 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" containerID="cri-o://f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937" gracePeriod=30 Nov 02 23:16:12.266077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.266042 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:12.343603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.343573 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m"] Nov 02 23:16:12.344285 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:12.344255 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb8be88_5fbf_45dc_8925_1fb51e6829e0.slice/crio-9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee WatchSource:0}: Error finding container 9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee: Status 404 returned error can't find the container with id 9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee Nov 02 23:16:12.733371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.733334 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" event={"ID":"feb8be88-5fbf-45dc-8925-1fb51e6829e0","Type":"ContainerStarted","Data":"e50e3cac944025909dc1d60867e10412f165d42a70c9f69dfc9182404dad46b5"} Nov 02 23:16:12.733541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.733395 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" event={"ID":"feb8be88-5fbf-45dc-8925-1fb51e6829e0","Type":"ContainerStarted","Data":"9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee"} Nov 02 23:16:12.850401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:12.850303 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" podStartSLOduration=1.850290165 podStartE2EDuration="1.850290165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:12.849775589 +0000 UTC m=+1687.279645450" watchObservedRunningTime="2026-11-02 23:16:12.850290165 +0000 UTC m=+1687.280160026" Nov 02 23:16:13.049841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.049808 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw"] Nov 02 23:16:13.053479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.053452 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef02a122-tzppw"] Nov 02 23:16:13.697257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.697219 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50650c51-3047-4ad6-85f9-631a06dbc683" path="/var/lib/kubelet/pods/50650c51-3047-4ad6-85f9-631a06dbc683/volumes" Nov 02 23:16:13.749984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.749952 4494 generic.go:356] "Generic (PLEG): container finished" podID="feb8be88-5fbf-45dc-8925-1fb51e6829e0" containerID="e50e3cac944025909dc1d60867e10412f165d42a70c9f69dfc9182404dad46b5" exitCode=0 Nov 02 23:16:13.750159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.750084 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" event={"ID":"feb8be88-5fbf-45dc-8925-1fb51e6829e0","Type":"ContainerDied","Data":"e50e3cac944025909dc1d60867e10412f165d42a70c9f69dfc9182404dad46b5"} Nov 02 23:16:13.998612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:13.998534 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:14.147896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.147856 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:14.392295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.378139 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:14.649351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.633241 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878454 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878501 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a2951b-332d-46d8-8db3-f0e97836fc8d" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878561 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878584 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3566791a-0d40-45f9-8f28-7319e88735db" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878611 4494 cpu_manager.go:414] "containerMap: removing container" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878619 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878909 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="917d95ef-ccde-4f71-b49d-4527ff30b9c1" containerName="cleanup" Nov 02 23:16:14.884194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.878933 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ea8701-ac08-45de-b25f-f2e8dce32514" containerName="cleanup" Nov 02 23:16:14.969419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.906459 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:16:14.969419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.906520 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:16:14.976590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:14.970793 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.007089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.006891 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.007089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.006944 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllc2\" (UniqueName: \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.012374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.009106 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.019408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.017593 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:16:15.054251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.054227 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a7a67875-dockercfg-w7m78\"" Nov 02 23:16:15.104086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.104055 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a7a67875\"" Nov 02 23:16:15.108128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.108099 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bb5\" (UniqueName: \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.108478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.108451 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.108541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.108503 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllc2\" (UniqueName: \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.108541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.108535 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.109222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.109166 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.166756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.166730 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:15.170594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.170569 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllc2\" (UniqueName: \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\") pod \"payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.209986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.209959 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\") pod \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " Nov 02 23:16:15.210138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210036 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\" (UniqueName: \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\") pod \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " Nov 02 23:16:15.210247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210131 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\") pod \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\" (UID: \"feb8be88-5fbf-45dc-8925-1fb51e6829e0\") " Nov 02 23:16:15.210399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210379 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca" pod "feb8be88-5fbf-45dc-8925-1fb51e6829e0" (UID: "feb8be88-5fbf-45dc-8925-1fb51e6829e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:15.210470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210452 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bb5\" (UniqueName: \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.210706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210680 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.210786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.210761 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:15.211378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.211356 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.212152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.212132 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j" pod "feb8be88-5fbf-45dc-8925-1fb51e6829e0" (UID: "feb8be88-5fbf-45dc-8925-1fb51e6829e0"). InnerVolumeSpecName "kube-api-access-qnl7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:15.212383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.212364 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca" pod "feb8be88-5fbf-45dc-8925-1fb51e6829e0" (UID: "feb8be88-5fbf-45dc-8925-1fb51e6829e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:15.272164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.272137 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bb5\" (UniqueName: \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\") pod \"payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.312017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.311982 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnl7j\" (UniqueName: \"kubernetes.io/projected/feb8be88-5fbf-45dc-8925-1fb51e6829e0-kube-api-access-qnl7j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:15.312017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.312013 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb8be88-5fbf-45dc-8925-1fb51e6829e0-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:15.339617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.339590 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.353976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.353948 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:15.744473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.744443 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:16:15.744677 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:15.744656 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4349e3d_47d8_425b_a6dd_e752c947793c.slice/crio-d81e4ff1d986251a2e96f79ebc74391989aa7f1f8667d675e5ed776b9fbb4f02 WatchSource:0}: Error finding container d81e4ff1d986251a2e96f79ebc74391989aa7f1f8667d675e5ed776b9fbb4f02: Status 404 returned error can't find the container with id d81e4ff1d986251a2e96f79ebc74391989aa7f1f8667d675e5ed776b9fbb4f02 Nov 02 23:16:15.793227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.793204 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:16:15.796051 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:15.796026 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f10ed3f_b5dc_408a_9754_e20e4f14214e.slice/crio-87d63545d03fc10938d0438e893f1fbe0d008d59d66568e2cb8939a165e6708d WatchSource:0}: Error finding container 87d63545d03fc10938d0438e893f1fbe0d008d59d66568e2cb8939a165e6708d: Status 404 returned error can't find the container with id 87d63545d03fc10938d0438e893f1fbe0d008d59d66568e2cb8939a165e6708d Nov 02 23:16:15.825775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.825745 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" event={"ID":"e4349e3d-47d8-425b-a6dd-e752c947793c","Type":"ContainerStarted","Data":"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075"} Nov 02 23:16:15.825875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.825808 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:15.825875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.825832 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" event={"ID":"e4349e3d-47d8-425b-a6dd-e752c947793c","Type":"ContainerStarted","Data":"d81e4ff1d986251a2e96f79ebc74391989aa7f1f8667d675e5ed776b9fbb4f02"} Nov 02 23:16:15.827054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.827028 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" event={"ID":"0f10ed3f-b5dc-408a-9754-e20e4f14214e","Type":"ContainerStarted","Data":"87d63545d03fc10938d0438e893f1fbe0d008d59d66568e2cb8939a165e6708d"} Nov 02 23:16:15.828292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.828273 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" Nov 02 23:16:15.828366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.828330 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m" event={"ID":"feb8be88-5fbf-45dc-8925-1fb51e6829e0","Type":"ContainerDied","Data":"9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee"} Nov 02 23:16:15.828401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.828368 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f262d8030ebaa97f9fd8d844df1dfd66a1252d1660b2df7ae225ef671c3e3ee" Nov 02 23:16:15.945878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:15.945826 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" podStartSLOduration=1.945812934 podStartE2EDuration="1.945812934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:15.941754847 +0000 UTC m=+1690.371624700" watchObservedRunningTime="2026-11-02 23:16:15.945812934 +0000 UTC m=+1690.375682795" Nov 02 23:16:16.712555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.712487 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:16:16.736345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736312 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736384 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736475 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736535 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736572 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736633 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736655 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:16.736809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736671 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736713 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736843 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\") pod \"a9f133f4-b997-4a5a-983f-4c52fddc641a\" (UID: \"a9f133f4-b997-4a5a-983f-4c52fddc641a\") " Nov 02 23:16:16.736989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736981 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:16.737113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.736986 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:16.737300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737129 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:16.737485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737460 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:16.737638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737616 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.737713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737643 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.737713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737659 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.737713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737673 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.737713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.737687 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f133f4-b997-4a5a-983f-4c52fddc641a-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.738966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.738932 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "kube-api-access-vz2l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:16.739120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.739099 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:16.739120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.739111 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:16.739961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.739947 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy" pod "a9f133f4-b997-4a5a-983f-4c52fddc641a" (UID: "a9f133f4-b997-4a5a-983f-4c52fddc641a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.839909 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2l2\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-kube-api-access-vz2l2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.839944 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.839959 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.839980 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f133f4-b997-4a5a-983f-4c52fddc641a-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.840439 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" event={"ID":"0f10ed3f-b5dc-408a-9754-e20e4f14214e","Type":"ContainerStarted","Data":"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868"} Nov 02 23:16:16.842451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.841378 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:16.845423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.845398 4494 generic.go:356] "Generic (PLEG): container finished" podID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerID="6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5" exitCode=0 Nov 02 23:16:16.845554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.845465 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" Nov 02 23:16:16.845554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.845485 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerDied","Data":"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5"} Nov 02 23:16:16.845554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.845549 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp" event={"ID":"a9f133f4-b997-4a5a-983f-4c52fddc641a","Type":"ContainerDied","Data":"20142dec93fbf338226122e370afdbc8fd375469b6ace61af1338b4352c78ced"} Nov 02 23:16:16.845669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.845577 4494 scope.go:122] "RemoveContainer" containerID="6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5" Nov 02 23:16:16.857796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.857778 4494 scope.go:122] "RemoveContainer" containerID="fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538" Nov 02 23:16:16.867943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.867794 4494 scope.go:122] "RemoveContainer" containerID="6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5" Nov 02 23:16:16.868091 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:16.868072 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5\": container with ID starting with 6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5 not found: ID does not exist" containerID="6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5" Nov 02 23:16:16.868144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.868100 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5"} err="failed to get container status \"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5\": rpc error: code = NotFound desc = could not find container \"6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5\": container with ID starting with 6009e8f451a2f571b1f87b7f99aeaa35c89737c7760fd9b8947a3be94fb902f5 not found: ID does not exist" Nov 02 23:16:16.868144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.868117 4494 scope.go:122] "RemoveContainer" containerID="fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538" Nov 02 23:16:16.868453 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:16.868432 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538\": container with ID starting with fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538 not found: ID does not exist" containerID="fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538" Nov 02 23:16:16.868523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:16.868462 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538"} err="failed to get container status \"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538\": rpc error: code = NotFound desc = could not find container \"fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538\": container with ID starting with fb4069222105a8a9f416e529e46187a7919a877eb4f6974ae22ef84f1f426538 not found: ID does not exist" Nov 02 23:16:17.008434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.008331 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" podStartSLOduration=3.008315198 podStartE2EDuration="3.008315198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:17.003741017 +0000 UTC m=+1691.433610880" watchObservedRunningTime="2026-11-02 23:16:17.008315198 +0000 UTC m=+1691.438185090" Nov 02 23:16:17.208271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.196057 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:16:17.218085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.218054 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8e34df-openshift-default-d5d8c7897-xwbdp"] Nov 02 23:16:17.416872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.416847 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:16:17.451071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451042 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451079 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451117 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451143 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451266 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451296 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451332 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451443 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451510 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\") pod \"723a6748-877b-43d4-ac76-0e14380a2cc3\" (UID: \"723a6748-877b-43d4-ac76-0e14380a2cc3\") " Nov 02 23:16:17.451667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451527 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:17.451667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451538 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:17.451871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451847 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:17.452091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451887 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:17.452091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.451971 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:17.452255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.452218 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.452255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.452237 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.452255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.452250 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.452427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.452264 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723a6748-877b-43d4-ac76-0e14380a2cc3-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.452427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.452280 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.453699 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.453674 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:17.453971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.453950 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:17.454303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.454283 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:17.454376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.454351 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2" pod "723a6748-877b-43d4-ac76-0e14380a2cc3" (UID: "723a6748-877b-43d4-ac76-0e14380a2cc3"). InnerVolumeSpecName "kube-api-access-776n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:17.553995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.553945 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776n2\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-kube-api-access-776n2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.553995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.553980 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.553995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.553993 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.554524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.554005 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723a6748-877b-43d4-ac76-0e14380a2cc3-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:17.584281 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:17.574798 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:16:17.695247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.695196 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" path="/var/lib/kubelet/pods/a9f133f4-b997-4a5a-983f-4c52fddc641a/volumes" Nov 02 23:16:17.854895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.854811 4494 generic.go:356] "Generic (PLEG): container finished" podID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerID="f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937" exitCode=0 Nov 02 23:16:17.854895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.854852 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" Nov 02 23:16:17.855516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.854892 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" event={"ID":"723a6748-877b-43d4-ac76-0e14380a2cc3","Type":"ContainerDied","Data":"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937"} Nov 02 23:16:17.855516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.854946 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j" event={"ID":"723a6748-877b-43d4-ac76-0e14380a2cc3","Type":"ContainerDied","Data":"09e1f8910ba0176b7d4f466898e873e12c79317a9326d140b8e54c98bebcf297"} Nov 02 23:16:17.855516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.854972 4494 scope.go:122] "RemoveContainer" containerID="f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937" Nov 02 23:16:17.867076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.867059 4494 scope.go:122] "RemoveContainer" containerID="f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937" Nov 02 23:16:17.867392 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:17.867369 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937\": container with ID starting with f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937 not found: ID does not exist" containerID="f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937" Nov 02 23:16:17.867495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:17.867396 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937"} err="failed to get container status \"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937\": rpc error: code = NotFound desc = could not find container \"f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937\": container with ID starting with f5779ca0e77fb29090a79c765a05248eca0a3f993a153d8a9e59e753d308c937 not found: ID does not exist" Nov 02 23:16:18.024136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:18.024105 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:16:18.074396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:18.074364 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-sc27j"] Nov 02 23:16:18.410723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:18.410689 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:18.885039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:18.884944 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:16:19.003460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:19.000204 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:16:19.248948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:19.248907 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:19.429381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:19.426351 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:19.694936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:19.694905 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" path="/var/lib/kubelet/pods/723a6748-877b-43d4-ac76-0e14380a2cc3/volumes" Nov 02 23:16:23.885644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:23.885569 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" podUID="1e8972bd-107f-4acf-9337-3acd07e4b553" containerName="payload-processing" containerID="cri-o://2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a" gracePeriod=25 Nov 02 23:16:24.042047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.040472 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" podUID="d7651354-c457-4bd4-adec-2812a16af7ac" containerName="payload-pre-processing" containerID="cri-o://cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb" gracePeriod=25 Nov 02 23:16:24.091794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.082306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:24.457221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.456674 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:16:24.496163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.496130 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\" (UniqueName: \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\") pod \"1e8972bd-107f-4acf-9337-3acd07e4b553\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " Nov 02 23:16:24.496401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.496231 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\") pod \"1e8972bd-107f-4acf-9337-3acd07e4b553\" (UID: \"1e8972bd-107f-4acf-9337-3acd07e4b553\") " Nov 02 23:16:24.496696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.496664 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume" pod "1e8972bd-107f-4acf-9337-3acd07e4b553" (UID: "1e8972bd-107f-4acf-9337-3acd07e4b553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:24.499040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.499012 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7" pod "1e8972bd-107f-4acf-9337-3acd07e4b553" (UID: "1e8972bd-107f-4acf-9337-3acd07e4b553"). InnerVolumeSpecName "kube-api-access-gx6b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:24.518551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.518528 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:16:24.597686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.597641 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\" (UniqueName: \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\") pod \"d7651354-c457-4bd4-adec-2812a16af7ac\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " Nov 02 23:16:24.597876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.597766 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\") pod \"d7651354-c457-4bd4-adec-2812a16af7ac\" (UID: \"d7651354-c457-4bd4-adec-2812a16af7ac\") " Nov 02 23:16:24.598275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.598241 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume" pod "d7651354-c457-4bd4-adec-2812a16af7ac" (UID: "d7651354-c457-4bd4-adec-2812a16af7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:24.598863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.598836 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7651354-c457-4bd4-adec-2812a16af7ac-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:24.598990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.598869 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6b7\" (UniqueName: \"kubernetes.io/projected/1e8972bd-107f-4acf-9337-3acd07e4b553-kube-api-access-gx6b7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:24.598990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.598886 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8972bd-107f-4acf-9337-3acd07e4b553-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:24.600164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.600132 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn" pod "d7651354-c457-4bd4-adec-2812a16af7ac" (UID: "d7651354-c457-4bd4-adec-2812a16af7ac"). InnerVolumeSpecName "kube-api-access-x4ghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:24.700964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.700932 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4ghn\" (UniqueName: \"kubernetes.io/projected/d7651354-c457-4bd4-adec-2812a16af7ac-kube-api-access-x4ghn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:24.914990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.914949 4494 generic.go:356] "Generic (PLEG): container finished" podID="d7651354-c457-4bd4-adec-2812a16af7ac" containerID="cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb" exitCode=0 Nov 02 23:16:24.915382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.915016 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" Nov 02 23:16:24.915382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.915034 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" event={"ID":"d7651354-c457-4bd4-adec-2812a16af7ac","Type":"ContainerDied","Data":"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb"} Nov 02 23:16:24.915382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.915093 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh" event={"ID":"d7651354-c457-4bd4-adec-2812a16af7ac","Type":"ContainerDied","Data":"fc0eba4243f1ae1fef7a077b6306d66d8e2048e2c339980f352c6bde143d8c54"} Nov 02 23:16:24.915382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.915116 4494 scope.go:122] "RemoveContainer" containerID="cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb" Nov 02 23:16:24.916769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.916746 4494 generic.go:356] "Generic (PLEG): container finished" podID="1e8972bd-107f-4acf-9337-3acd07e4b553" containerID="2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a" exitCode=0 Nov 02 23:16:24.916864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.916795 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" Nov 02 23:16:24.916921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.916792 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" event={"ID":"1e8972bd-107f-4acf-9337-3acd07e4b553","Type":"ContainerDied","Data":"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a"} Nov 02 23:16:24.916985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.916935 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd" event={"ID":"1e8972bd-107f-4acf-9337-3acd07e4b553","Type":"ContainerDied","Data":"375eec72319102380e4f898152c007be68d2dbde71d852775e1e58e4334bb177"} Nov 02 23:16:24.929006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.928980 4494 scope.go:122] "RemoveContainer" containerID="cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb" Nov 02 23:16:24.929400 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:24.929363 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb\": container with ID starting with cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb not found: ID does not exist" containerID="cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb" Nov 02 23:16:24.929560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.929530 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb"} err="failed to get container status \"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb\": rpc error: code = NotFound desc = could not find container \"cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb\": container with ID starting with cc8116e2b4667b45149925cefea522efdb4fff1d4e5a9d051d40931b4df17ddb not found: ID does not exist" Nov 02 23:16:24.929560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.929560 4494 scope.go:122] "RemoveContainer" containerID="2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a" Nov 02 23:16:24.965865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.965845 4494 scope.go:122] "RemoveContainer" containerID="2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a" Nov 02 23:16:24.966250 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:24.966220 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a\": container with ID starting with 2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a not found: ID does not exist" containerID="2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a" Nov 02 23:16:24.966355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.966258 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a"} err="failed to get container status \"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a\": rpc error: code = NotFound desc = could not find container \"2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a\": container with ID starting with 2d1109562d257027d512a70bc1f6f87eef96fc2d38d70972ddbab049159b942a not found: ID does not exist" Nov 02 23:16:24.989100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:24.989049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:25.145712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.145638 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:16:25.177230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.177154 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8ab111-7b4dcdcd59-fnzlh"] Nov 02 23:16:25.213681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.213605 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:16:25.217700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.217678 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ab111-6588dd75bf-p5xkd"] Nov 02 23:16:25.694556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.694527 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8972bd-107f-4acf-9337-3acd07e4b553" path="/var/lib/kubelet/pods/1e8972bd-107f-4acf-9337-3acd07e4b553/volumes" Nov 02 23:16:25.694998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:25.694982 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7651354-c457-4bd4-adec-2812a16af7ac" path="/var/lib/kubelet/pods/d7651354-c457-4bd4-adec-2812a16af7ac/volumes" Nov 02 23:16:26.862518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:26.862487 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:16:26.960674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:26.960641 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:27.269085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:27.269056 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:16:27.269315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:27.269292 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" containerID="cri-o://d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563" gracePeriod=30 Nov 02 23:16:27.320267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:27.320235 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:16:27.320450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:27.320431 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" containerID="cri-o://9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c" gracePeriod=30 Nov 02 23:16:27.895989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:27.895951 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:28.869466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:28.869437 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:16:29.256312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:29.256266 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:29.386509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:29.386467 4494 patch_prober.go:29] interesting pod/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:16:29.386724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:29.386574 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:16:30.586229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:30.586117 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:32.221748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.221708 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:32.674385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.674348 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:16:32.709052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.709027 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778718 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778773 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778840 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778867 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778942 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.778986 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779034 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779083 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779113 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779282 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779314 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779423 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779452 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779516 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779571 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779611 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779656 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\") pod \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\" (UID: \"09424ab0-6c4b-4b2f-9369-0663f13c3ac4\") " Nov 02 23:16:32.805988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.779702 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\") pod \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\" (UID: \"2107d46e-3bab-48f9-924a-ba2d841bfd4a\") " Nov 02 23:16:32.840200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.820201 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:32.840749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.840459 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.841355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.840943 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.843728 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.843764 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.843786 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.847401 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.847584 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.856987 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.863310 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.863370 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.863484 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.863698 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.863952 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.864097 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.867041 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.868475 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.870386 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.874493 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.876680 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "kube-api-access-c682r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.877087 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59" pod "2107d46e-3bab-48f9-924a-ba2d841bfd4a" (UID: "2107d46e-3bab-48f9-924a-ba2d841bfd4a"). InnerVolumeSpecName "kube-api-access-nrt59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:32.903357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.903286 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket" pod "09424ab0-6c4b-4b2f-9369-0663f13c3ac4" (UID: "09424ab0-6c4b-4b2f-9369-0663f13c3ac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.937248 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938598 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1e8972bd-107f-4acf-9337-3acd07e4b553" containerName="payload-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938621 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8972bd-107f-4acf-9337-3acd07e4b553" containerName="payload-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938649 4494 cpu_manager.go:414] "containerMap: removing container" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938656 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938672 4494 cpu_manager.go:414] "containerMap: removing container" podUID="feb8be88-5fbf-45dc-8925-1fb51e6829e0" containerName="revoke-keys" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938683 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb8be88-5fbf-45dc-8925-1fb51e6829e0" containerName="revoke-keys" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938704 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938712 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938731 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938739 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938759 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d7651354-c457-4bd4-adec-2812a16af7ac" containerName="payload-pre-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938767 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7651354-c457-4bd4-adec-2812a16af7ac" containerName="payload-pre-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938791 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938798 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938822 4494 cpu_manager.go:414] "containerMap: removing container" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938830 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938846 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.938853 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939012 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939029 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="723a6748-877b-43d4-ac76-0e14380a2cc3" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939046 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8972bd-107f-4acf-9337-3acd07e4b553" containerName="payload-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939058 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939068 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb8be88-5fbf-45dc-8925-1fb51e6829e0" containerName="revoke-keys" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939078 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939091 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerName="istio-proxy" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.939103 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7651354-c457-4bd4-adec-2812a16af7ac" containerName="payload-pre-processing" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956473 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956506 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956522 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956537 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c682r\" (UniqueName: \"kubernetes.io/projected/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-kube-api-access-c682r\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956552 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956564 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956575 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956587 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2107d46e-3bab-48f9-924a-ba2d841bfd4a-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956602 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956615 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956630 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956644 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrt59\" (UniqueName: \"kubernetes.io/projected/2107d46e-3bab-48f9-924a-ba2d841bfd4a-kube-api-access-nrt59\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956657 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956703 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09424ab0-6c4b-4b2f-9369-0663f13c3ac4-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:32.956719 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2107d46e-3bab-48f9-924a-ba2d841bfd4a-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.013338 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:16:33.040602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.013474 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.062998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063140 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063232 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zt47\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063276 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063302 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063365 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063398 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063475 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.077990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.063508 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.081394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.080321 4494 generic.go:356] "Generic (PLEG): container finished" podID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" containerID="9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c" exitCode=0 Nov 02 23:16:33.081394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.080550 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerDied","Data":"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c"} Nov 02 23:16:33.081394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.080608 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" event={"ID":"2107d46e-3bab-48f9-924a-ba2d841bfd4a","Type":"ContainerDied","Data":"cde00341d658c366e677c3b8a178ef2e26ae967a729bd40de5deeaa3a1d0ad8f"} Nov 02 23:16:33.081394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.080643 4494 scope.go:122] "RemoveContainer" containerID="9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c" Nov 02 23:16:33.081394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.080838 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.201019 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.201091 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zt47\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.201136 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.204205 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.204321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.204365 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.204451 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.220238 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.228806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.220418 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.238396 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.238991 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.242348 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.242623 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.243022 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.243296 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.257017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.243475 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.263960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.263921 4494 generic.go:356] "Generic (PLEG): container finished" podID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" containerID="d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563" exitCode=0 Nov 02 23:16:33.264141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.264124 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" event={"ID":"09424ab0-6c4b-4b2f-9369-0663f13c3ac4","Type":"ContainerDied","Data":"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563"} Nov 02 23:16:33.264263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.264226 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" event={"ID":"09424ab0-6c4b-4b2f-9369-0663f13c3ac4","Type":"ContainerDied","Data":"d6e1a2b51b3d79527f72398f946a69dc61a8eef0781ab6effec054970681ef9f"} Nov 02 23:16:33.264367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.264342 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6" Nov 02 23:16:33.265609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.265584 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-aa2454-openshift-default-dockercfg-q2hz4\"" Nov 02 23:16:33.295938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.295909 4494 scope.go:122] "RemoveContainer" containerID="1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b" Nov 02 23:16:33.320496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.320473 4494 scope.go:122] "RemoveContainer" containerID="9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c" Nov 02 23:16:33.320886 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:33.320854 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c\": container with ID starting with 9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c not found: ID does not exist" containerID="9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c" Nov 02 23:16:33.320978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.320901 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c"} err="failed to get container status \"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c\": rpc error: code = NotFound desc = could not find container \"9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c\": container with ID starting with 9538279edfbb42e21b14936c073fca6d7196b4f2a97831b1797305b0551d7e5c not found: ID does not exist" Nov 02 23:16:33.320978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.320927 4494 scope.go:122] "RemoveContainer" containerID="1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b" Nov 02 23:16:33.321298 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:33.321267 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b\": container with ID starting with 1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b not found: ID does not exist" containerID="1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b" Nov 02 23:16:33.321396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.321302 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b"} err="failed to get container status \"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b\": rpc error: code = NotFound desc = could not find container \"1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b\": container with ID starting with 1f33f2b0c242cf9b5d27a400a4a6a94261c6e998aa2acde63d0e87543ca3217b not found: ID does not exist" Nov 02 23:16:33.321396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.321325 4494 scope.go:122] "RemoveContainer" containerID="d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563" Nov 02 23:16:33.329568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.329501 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.329779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.329748 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zt47\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\") pod \"e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.333263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.333236 4494 scope.go:122] "RemoveContainer" containerID="d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563" Nov 02 23:16:33.333617 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:33.333591 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563\": container with ID starting with d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563 not found: ID does not exist" containerID="d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563" Nov 02 23:16:33.333781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.333626 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563"} err="failed to get container status \"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563\": rpc error: code = NotFound desc = could not find container \"d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563\": container with ID starting with d6a0abdc509a1492b34ca3daa9dd17c143c2079957ae787e233dbd2c6974c563 not found: ID does not exist" Nov 02 23:16:33.380010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.379976 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:33.589257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.576096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:33.633686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.633649 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:16:33.638091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.638065 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-8dpc6"] Nov 02 23:16:33.696136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.696103 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09424ab0-6c4b-4b2f-9369-0663f13c3ac4" path="/var/lib/kubelet/pods/09424ab0-6c4b-4b2f-9369-0663f13c3ac4/volumes" Nov 02 23:16:33.696985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.696961 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:16:33.700197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.700157 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-7nsgp"] Nov 02 23:16:33.815878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.815845 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:16:33.816656 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:33.816619 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9fb997_5eec_4d00_b7eb_0dfbdb3f147f.slice/crio-0d9596022c3531dfb99ea7cce0e126a93ef5cf8685a4fc115d27cc9b19e4e9c6 WatchSource:0}: Error finding container 0d9596022c3531dfb99ea7cce0e126a93ef5cf8685a4fc115d27cc9b19e4e9c6: Status 404 returned error can't find the container with id 0d9596022c3531dfb99ea7cce0e126a93ef5cf8685a4fc115d27cc9b19e4e9c6 Nov 02 23:16:33.819655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.819622 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:33.819781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.819680 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:33.819781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:33.819710 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:34.273973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.273924 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerStarted","Data":"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f"} Nov 02 23:16:34.274472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.273993 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerStarted","Data":"0d9596022c3531dfb99ea7cce0e126a93ef5cf8685a4fc115d27cc9b19e4e9c6"} Nov 02 23:16:34.372458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.372221 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": read tcp 10.133.0.2:43480->10.133.0.120:15021: read: connection reset by peer" start-of-body= Nov 02 23:16:34.372458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.372285 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": read tcp 10.133.0.2:43480->10.133.0.120:15021: read: connection reset by peer" Nov 02 23:16:34.380415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.380389 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:34.387202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.387147 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:34.399496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:34.399447 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podStartSLOduration=2.399432072 podStartE2EDuration="2.399432072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:34.39545634 +0000 UTC m=+1708.825326202" watchObservedRunningTime="2026-11-02 23:16:34.399432072 +0000 UTC m=+1708.829301933" Nov 02 23:16:35.033009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.032961 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:35.177621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.177586 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:35.284810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.284727 4494 generic.go:356] "Generic (PLEG): container finished" podID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerID="56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10" exitCode=0 Nov 02 23:16:35.285262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.284807 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerDied","Data":"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10"} Nov 02 23:16:35.285262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.285115 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:35.285585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.285567 4494 scope.go:122] "RemoveContainer" containerID="56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10" Nov 02 23:16:35.286091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.286073 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:16:35.287200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.287151 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:35.287277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.287232 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:35.287277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.287265 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:35.707087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.707055 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2107d46e-3bab-48f9-924a-ba2d841bfd4a" path="/var/lib/kubelet/pods/2107d46e-3bab-48f9-924a-ba2d841bfd4a/volumes" Nov 02 23:16:35.855396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:35.855363 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:36.101860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.101766 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:36.175747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.174390 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:16:36.295960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.295928 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerStarted","Data":"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d"} Nov 02 23:16:36.373501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.373342 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:16:36.375603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.374677 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f133f4-b997-4a5a-983f-4c52fddc641a" containerName="istio-proxy" Nov 02 23:16:36.378489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.378318 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.404751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.404693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:36.465979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.465938 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:16:36.473619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.473590 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-048ef3-openshift-default-dockercfg-8ffs8\"" Nov 02 23:16:36.496199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496147 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496214 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496265 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496290 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496327 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496349 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cnc\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496408 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496462 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.496718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.496490 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.558451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.558401 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:36.611210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.611144 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.611524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.611499 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.611727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.611706 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.611884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.611864 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.612050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.612034 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.612208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.612191 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cnc\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.612436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.612417 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.612621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.612605 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.612769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.612751 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.628944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.613933 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.629316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.614526 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.629524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.615098 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.629673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.615361 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.629797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.615584 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.681258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.677162 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.717197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.714464 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.727084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.727050 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.747039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.746948 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cnc\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\") pod \"e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:36.894774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.894682 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:16:36.896203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.895056 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:36.896203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:36.895234 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:37.004569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.004543 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:37.198816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.198740 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:37.472080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.471857 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:37.586431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.586391 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:16:37.645079 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:37.645046 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e395c83_78c6_4ea4_8d62_ba136f8f3e7b.slice/crio-ea1c4cf96a9b8c9fc3788fd47859e4488ac8946a44a6ba5ac29ade9f93a06183 WatchSource:0}: Error finding container ea1c4cf96a9b8c9fc3788fd47859e4488ac8946a44a6ba5ac29ade9f93a06183: Status 404 returned error can't find the container with id ea1c4cf96a9b8c9fc3788fd47859e4488ac8946a44a6ba5ac29ade9f93a06183 Nov 02 23:16:37.653821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.652303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:37.679231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.655580 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:37.679231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.655680 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:37.679231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.655726 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:16:37.911201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.887675 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:37.911201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:37.887755 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:38.140211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.138562 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:16:38.140211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.138895 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" containerID="cri-o://362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d" gracePeriod=30 Nov 02 23:16:38.174341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.173838 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:16:38.174341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.174105 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" containerID="cri-o://c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d" gracePeriod=30 Nov 02 23:16:38.194921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.194785 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:16:38.196288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.196159 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" containerID="cri-o://d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a" gracePeriod=30 Nov 02 23:16:38.206483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.206456 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:16:38.207123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.207091 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" containerID="cri-o://a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5" gracePeriod=30 Nov 02 23:16:38.319000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.318961 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerStarted","Data":"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3"} Nov 02 23:16:38.319187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.319020 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerStarted","Data":"ea1c4cf96a9b8c9fc3788fd47859e4488ac8946a44a6ba5ac29ade9f93a06183"} Nov 02 23:16:38.797780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.797716 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podStartSLOduration=2.797698487 podStartE2EDuration="2.797698487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:38.796882235 +0000 UTC m=+1713.226752097" watchObservedRunningTime="2026-11-02 23:16:38.797698487 +0000 UTC m=+1713.227568347" Nov 02 23:16:38.885281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.885247 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:38.885440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:38.885305 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:39.022246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.017109 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:39.034196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.028496 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:39.326346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.326320 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:39.327288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.327272 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:16:39.885471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.885430 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:39.885970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:39.885489 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:40.552391 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.552314 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:16:40.552697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.552663 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:16:40.765197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.757273 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:40.819313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.806811 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:16:40.886693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.886660 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:40.887153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.886721 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:40.923426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:40.923389 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:41.175704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.175652 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" podUID="3539f21f-6d47-4226-8da5-e7d28b270bd2" containerName="payload-pre-processing" containerID="cri-o://4bd52c4d142fab129bef441aff5454c3216321613623b913978c2d2885b7090a" gracePeriod=25 Nov 02 23:16:41.261208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.251121 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:16:41.270534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.269455 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.327595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.327228 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.327595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.327379 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8d8g\" (UniqueName: \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.328355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.328332 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-aa2454\"" Nov 02 23:16:41.329802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.329781 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-aa2454-dockercfg-hd8fw\"" Nov 02 23:16:41.345199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.332387 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:16:41.463805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.463702 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.464320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.464275 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8d8g\" (UniqueName: \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.466254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.466227 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.728798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.728702 4494 generic.go:356] "Generic (PLEG): container finished" podID="3539f21f-6d47-4226-8da5-e7d28b270bd2" containerID="4bd52c4d142fab129bef441aff5454c3216321613623b913978c2d2885b7090a" exitCode=0 Nov 02 23:16:41.743388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.743331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:41.766369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.766322 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" event={"ID":"3539f21f-6d47-4226-8da5-e7d28b270bd2","Type":"ContainerDied","Data":"4bd52c4d142fab129bef441aff5454c3216321613623b913978c2d2885b7090a"} Nov 02 23:16:41.766684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.766661 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:16:41.854250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.854157 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8d8g\" (UniqueName: \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\") pod \"payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.872980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.872947 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:16:41.875555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.875438 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:41.892710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.892122 4494 patch_prober.go:29] interesting pod/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:16:41.892710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.892196 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 23:16:41.924410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.924373 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:41.967695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.967619 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:16:41.968160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:41.968108 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:16:42.013062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.012709 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rsx\" (UniqueName: \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.013062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.012823 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.099634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.099610 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:16:42.114700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.113865 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rsx\" (UniqueName: \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.114700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.113940 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.114700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.114665 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.175997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.175966 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rsx\" (UniqueName: \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.205825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.205791 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:42.215057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.215026 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\") pod \"3539f21f-6d47-4226-8da5-e7d28b270bd2\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " Nov 02 23:16:42.215251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.215234 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\" (UniqueName: \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\") pod \"3539f21f-6d47-4226-8da5-e7d28b270bd2\" (UID: \"3539f21f-6d47-4226-8da5-e7d28b270bd2\") " Nov 02 23:16:42.215482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.215456 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume" pod "3539f21f-6d47-4226-8da5-e7d28b270bd2" (UID: "3539f21f-6d47-4226-8da5-e7d28b270bd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:42.215786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.215750 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3539f21f-6d47-4226-8da5-e7d28b270bd2-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:42.218272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.218135 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl" pod "3539f21f-6d47-4226-8da5-e7d28b270bd2" (UID: "3539f21f-6d47-4226-8da5-e7d28b270bd2"). InnerVolumeSpecName "kube-api-access-m8dpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:42.317294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.317144 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dpl\" (UniqueName: \"kubernetes.io/projected/3539f21f-6d47-4226-8da5-e7d28b270bd2-kube-api-access-m8dpl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:42.371754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.371710 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:16:42.470435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.470408 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:42.627831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.627751 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:42.688189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.688145 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:16:42.688341 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:42.688303 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf2becf_9f2d_426d_b0fd_b618d1ce6850.slice/crio-090c775d4dc8c75b6b89f12e8d48e95f9b117a04f1970d5b234b1c6bc558e41a WatchSource:0}: Error finding container 090c775d4dc8c75b6b89f12e8d48e95f9b117a04f1970d5b234b1c6bc558e41a: Status 404 returned error can't find the container with id 090c775d4dc8c75b6b89f12e8d48e95f9b117a04f1970d5b234b1c6bc558e41a Nov 02 23:16:42.801802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.801289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:42.826301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.820319 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" event={"ID":"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f","Type":"ContainerStarted","Data":"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c"} Nov 02 23:16:42.826301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.820402 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" event={"ID":"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f","Type":"ContainerStarted","Data":"63a8fa9da5f348a28f8bfee5e46a964ff96a23bf37f1b0c5fad8a11f3085f1a6"} Nov 02 23:16:42.826301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.821689 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:42.847516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.847480 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" event={"ID":"ecf2becf-9f2d-426d-b0fd-b618d1ce6850","Type":"ContainerStarted","Data":"090c775d4dc8c75b6b89f12e8d48e95f9b117a04f1970d5b234b1c6bc558e41a"} Nov 02 23:16:42.853951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.853917 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" event={"ID":"3539f21f-6d47-4226-8da5-e7d28b270bd2","Type":"ContainerDied","Data":"34404673f662fa2a10749686b55c87b8abe3286d024bc1ed7ac94d3c58a1fd68"} Nov 02 23:16:42.854299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.854282 4494 scope.go:122] "RemoveContainer" containerID="4bd52c4d142fab129bef441aff5454c3216321613623b913978c2d2885b7090a" Nov 02 23:16:42.854693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.854677 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt" Nov 02 23:16:42.975817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:42.975599 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" podStartSLOduration=1.975577318 podStartE2EDuration="1.975577318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:42.942277283 +0000 UTC m=+1717.372147145" watchObservedRunningTime="2026-11-02 23:16:42.975577318 +0000 UTC m=+1717.405447181" Nov 02 23:16:43.047533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.047501 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:16:43.048964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.048937 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:16:43.049563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.049539 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:16:43.062928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.062894 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:16:43.091845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.091763 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5153fe-64c78444c5-ghdpt"] Nov 02 23:16:43.159301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.151318 4494 patch_prober.go:29] interesting pod/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:16:43.159301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.151367 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:16:43.508519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.508072 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:16:43.568688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.568666 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:16:43.582896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.582870 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:16:43.588061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588042 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588073 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588213 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588286 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.588376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588307 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588338 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588386 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.588631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588607 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.589105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588860 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.589105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.588906 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.589105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.589014 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\") pod \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\" (UID: \"e64f1ca3-5ad8-4e42-b1da-134c377aacdc\") " Nov 02 23:16:43.589367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.589231 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.589367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.589285 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.589512 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590373 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590400 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590418 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590434 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.590479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590452 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.590943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590896 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:43.591014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.590985 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.591081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.591018 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "kube-api-access-58s8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.592038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.592012 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy" pod "e64f1ca3-5ad8-4e42-b1da-134c377aacdc" (UID: "e64f1ca3-5ad8-4e42-b1da-134c377aacdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.593560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.593539 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:16:43.691470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691419 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.691470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691474 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.691757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691512 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.691757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691568 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.691757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691603 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.691757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691650 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.691757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691690 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691759 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691813 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691864 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691914 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.691957 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692001 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692015 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.692086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692055 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692090 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692128 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692186 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692234 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692277 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692304 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692350 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692369 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692399 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\") pod \"4625b18a-a96b-4208-b22d-cc49548112a0\" (UID: \"4625b18a-a96b-4208-b22d-cc49548112a0\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692439 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692494 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\") pod \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\" (UID: \"931eb30c-36be-49cf-8b2f-25f5c5bf1497\") " Nov 02 23:16:43.692578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692523 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.693323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692594 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.693323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692626 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:43.693323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692646 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\") pod \"10072372-541a-468f-971f-6c60c604415e\" (UID: \"10072372-541a-468f-971f-6c60c604415e\") " Nov 02 23:16:43.693323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.692650 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.693323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693265 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693496 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4625b18a-a96b-4208-b22d-cc49548112a0-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693518 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693552 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693570 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693585 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693599 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693656 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693674 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693689 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s8h\" (UniqueName: \"kubernetes.io/projected/e64f1ca3-5ad8-4e42-b1da-134c377aacdc-kube-api-access-58s8h\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.693834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693693 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.694033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.693848 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.694467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.694395 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.694695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.694582 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.694767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.694713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.695783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.695515 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:43.697165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.696028 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.697165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.696164 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.697165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.696905 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:43.697401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.697199 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.697458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.697397 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.698230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.697599 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.698230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.697765 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:43.698230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.697846 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:43.698825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.698796 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "kube-api-access-fv879". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.698941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.698818 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "kube-api-access-c6hlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.698941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.698890 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.698941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.698893 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.699068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.698945 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy" pod "4625b18a-a96b-4208-b22d-cc49548112a0" (UID: "4625b18a-a96b-4208-b22d-cc49548112a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:16:43.699706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.699687 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:16:43.699997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.699969 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz" pod "931eb30c-36be-49cf-8b2f-25f5c5bf1497" (UID: "931eb30c-36be-49cf-8b2f-25f5c5bf1497"). InnerVolumeSpecName "kube-api-access-42ntz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.700537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.700517 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token" pod "10072372-541a-468f-971f-6c60c604415e" (UID: "10072372-541a-468f-971f-6c60c604415e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.701329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.701304 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3539f21f-6d47-4226-8da5-e7d28b270bd2" path="/var/lib/kubelet/pods/3539f21f-6d47-4226-8da5-e7d28b270bd2/volumes" Nov 02 23:16:43.795312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795282 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ntz\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-kube-api-access-42ntz\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795309 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795322 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795330 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10072372-541a-468f-971f-6c60c604415e-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795351 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795360 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795370 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hlg\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-kube-api-access-c6hlg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795378 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4625b18a-a96b-4208-b22d-cc49548112a0-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795386 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795395 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10072372-541a-468f-971f-6c60c604415e-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795403 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10072372-541a-468f-971f-6c60c604415e-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795415 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4625b18a-a96b-4208-b22d-cc49548112a0-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795428 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795439 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv879\" (UniqueName: \"kubernetes.io/projected/10072372-541a-468f-971f-6c60c604415e-kube-api-access-fv879\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795448 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795457 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795465 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795483 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795491 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795499 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931eb30c-36be-49cf-8b2f-25f5c5bf1497-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795506 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.795565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.795514 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4625b18a-a96b-4208-b22d-cc49548112a0-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:16:43.862129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.862093 4494 generic.go:356] "Generic (PLEG): container finished" podID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerID="362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d" exitCode=0 Nov 02 23:16:43.862351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.862144 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" Nov 02 23:16:43.862351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.862209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" event={"ID":"e64f1ca3-5ad8-4e42-b1da-134c377aacdc","Type":"ContainerDied","Data":"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d"} Nov 02 23:16:43.862351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.862255 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr" event={"ID":"e64f1ca3-5ad8-4e42-b1da-134c377aacdc","Type":"ContainerDied","Data":"13dd18d73f89a0b3ae76eb5cf64b4535409698226b6b1c82a033d1a00ccc3920"} Nov 02 23:16:43.862351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.862282 4494 scope.go:122] "RemoveContainer" containerID="362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d" Nov 02 23:16:43.864547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.864438 4494 generic.go:356] "Generic (PLEG): container finished" podID="4625b18a-a96b-4208-b22d-cc49548112a0" containerID="a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5" exitCode=0 Nov 02 23:16:43.864547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.864511 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" Nov 02 23:16:43.864756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.864544 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerDied","Data":"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5"} Nov 02 23:16:43.864756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.864589 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f" event={"ID":"4625b18a-a96b-4208-b22d-cc49548112a0","Type":"ContainerDied","Data":"d9c360bd85cf487de52ab3cde767f7f7db48ef1dee3b85c231e40f3346a48039"} Nov 02 23:16:43.866551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.866528 4494 generic.go:356] "Generic (PLEG): container finished" podID="10072372-541a-468f-971f-6c60c604415e" containerID="d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a" exitCode=0 Nov 02 23:16:43.866859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.866580 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" Nov 02 23:16:43.866859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.866680 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" event={"ID":"10072372-541a-468f-971f-6c60c604415e","Type":"ContainerDied","Data":"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a"} Nov 02 23:16:43.866859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.866720 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2" event={"ID":"10072372-541a-468f-971f-6c60c604415e","Type":"ContainerDied","Data":"591d840a8c6a8c4f4dabf57c8247c098f3ff9f26cabf5cb8e58cf8fad63b0f51"} Nov 02 23:16:43.868437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.868415 4494 generic.go:356] "Generic (PLEG): container finished" podID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerID="c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d" exitCode=0 Nov 02 23:16:43.868571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.868483 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" event={"ID":"931eb30c-36be-49cf-8b2f-25f5c5bf1497","Type":"ContainerDied","Data":"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d"} Nov 02 23:16:43.868571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.868534 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" event={"ID":"931eb30c-36be-49cf-8b2f-25f5c5bf1497","Type":"ContainerDied","Data":"43e176dc32c4aa7e5fb828ec3d0f10595be1faea3f714da7083314c5741600cd"} Nov 02 23:16:43.870453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.870432 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" event={"ID":"ecf2becf-9f2d-426d-b0fd-b618d1ce6850","Type":"ContainerStarted","Data":"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc"} Nov 02 23:16:43.872561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.872544 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:43.872695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.872681 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn" Nov 02 23:16:43.875368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.875346 4494 scope.go:122] "RemoveContainer" containerID="362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d" Nov 02 23:16:43.875620 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:43.875601 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d\": container with ID starting with 362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d not found: ID does not exist" containerID="362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d" Nov 02 23:16:43.875694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.875627 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d"} err="failed to get container status \"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d\": rpc error: code = NotFound desc = could not find container \"362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d\": container with ID starting with 362a3aaf430f8d8010b5bb831ac19ea10ff306c57e9ba28196d7ce261a66d68d not found: ID does not exist" Nov 02 23:16:43.875694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.875646 4494 scope.go:122] "RemoveContainer" containerID="a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5" Nov 02 23:16:43.885134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.885119 4494 scope.go:122] "RemoveContainer" containerID="73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d" Nov 02 23:16:43.914667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.914635 4494 scope.go:122] "RemoveContainer" containerID="a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5" Nov 02 23:16:43.915117 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:43.915082 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5\": container with ID starting with a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5 not found: ID does not exist" containerID="a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5" Nov 02 23:16:43.915290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.915127 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5"} err="failed to get container status \"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5\": rpc error: code = NotFound desc = could not find container \"a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5\": container with ID starting with a33e2b71c108af9f38db9e630f1817c37481ba8783a6ddfa1873ba1427287bd5 not found: ID does not exist" Nov 02 23:16:43.915290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.915156 4494 scope.go:122] "RemoveContainer" containerID="73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d" Nov 02 23:16:43.915703 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:43.915668 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d\": container with ID starting with 73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d not found: ID does not exist" containerID="73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d" Nov 02 23:16:43.915813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.915712 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d"} err="failed to get container status \"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d\": rpc error: code = NotFound desc = could not find container \"73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d\": container with ID starting with 73e230dbbac35489b2b9d1854bda7a627e61f6de830bb359cb2973ab6b539d8d not found: ID does not exist" Nov 02 23:16:43.915813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.915742 4494 scope.go:122] "RemoveContainer" containerID="d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a" Nov 02 23:16:43.930968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.930945 4494 scope.go:122] "RemoveContainer" containerID="d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a" Nov 02 23:16:43.931390 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:43.931363 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a\": container with ID starting with d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a not found: ID does not exist" containerID="d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a" Nov 02 23:16:43.931486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.931401 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a"} err="failed to get container status \"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a\": rpc error: code = NotFound desc = could not find container \"d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a\": container with ID starting with d281c517caf8b5164b9188f5f214ef318d8a1b004170576eeaf023185862fb7a not found: ID does not exist" Nov 02 23:16:43.931486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.931428 4494 scope.go:122] "RemoveContainer" containerID="c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d" Nov 02 23:16:43.947280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.947255 4494 scope.go:122] "RemoveContainer" containerID="c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d" Nov 02 23:16:43.947564 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:43.947540 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d\": container with ID starting with c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d not found: ID does not exist" containerID="c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d" Nov 02 23:16:43.947682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:43.947573 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d"} err="failed to get container status \"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d\": rpc error: code = NotFound desc = could not find container \"c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d\": container with ID starting with c3e906fd8773e970e8ca43cad7f93b927e70459e92ddc55797511531df57968d not found: ID does not exist" Nov 02 23:16:44.160647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.160618 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:16:44.164665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.164639 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-mh9z2"] Nov 02 23:16:44.276218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.276163 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:16:44.279815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.279789 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-jsbcn"] Nov 02 23:16:44.395512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.395461 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" podStartSLOduration=3.395446764 podStartE2EDuration="3.395446764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:44.392697938 +0000 UTC m=+1718.822567797" watchObservedRunningTime="2026-11-02 23:16:44.395446764 +0000 UTC m=+1718.825316625" Nov 02 23:16:44.558262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.554242 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:16:44.570984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.568515 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-vpf5f"] Nov 02 23:16:44.667274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.667240 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:16:44.671468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:44.671438 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-637fdb-openshift-default-5647849b9-nj2dr"] Nov 02 23:16:45.671215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.671149 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:45.695021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.694982 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10072372-541a-468f-971f-6c60c604415e" path="/var/lib/kubelet/pods/10072372-541a-468f-971f-6c60c604415e/volumes" Nov 02 23:16:45.695667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.695643 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" path="/var/lib/kubelet/pods/4625b18a-a96b-4208-b22d-cc49548112a0/volumes" Nov 02 23:16:45.696288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.696268 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" path="/var/lib/kubelet/pods/931eb30c-36be-49cf-8b2f-25f5c5bf1497/volumes" Nov 02 23:16:45.697186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.697146 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" path="/var/lib/kubelet/pods/e64f1ca3-5ad8-4e42-b1da-134c377aacdc/volumes" Nov 02 23:16:45.921437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:45.920412 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:46.081287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.081253 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:16:46.081958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.081940 4494 cpu_manager.go:414] "containerMap: removing container" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.081961 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082003 4494 cpu_manager.go:414] "containerMap: removing container" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082011 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082024 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082030 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082043 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082047 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082055 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082060 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082069 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3539f21f-6d47-4226-8da5-e7d28b270bd2" containerName="payload-pre-processing" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082074 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3539f21f-6d47-4226-8da5-e7d28b270bd2" containerName="payload-pre-processing" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082195 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="931eb30c-36be-49cf-8b2f-25f5c5bf1497" containerName="istio-proxy" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082203 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64f1ca3-5ad8-4e42-b1da-134c377aacdc" containerName="istio-proxy" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082213 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082220 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4625b18a-a96b-4208-b22d-cc49548112a0" containerName="istio-proxy" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082226 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3539f21f-6d47-4226-8da5-e7d28b270bd2" containerName="payload-pre-processing" Nov 02 23:16:46.082426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.082237 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="10072372-541a-468f-971f-6c60c604415e" containerName="istio-proxy" Nov 02 23:16:46.085626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.085606 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.142948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.142917 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:16:46.213297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.208738 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-048ef3-dockercfg-t6q7w\"" Nov 02 23:16:46.213297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.208860 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-048ef3\"" Nov 02 23:16:46.252609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.247130 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcsl\" (UniqueName: \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.252609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.247411 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.362269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.339308 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:16:46.362269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.360290 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcsl\" (UniqueName: \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.362269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.360486 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.362269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.361538 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.375016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.363851 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.407315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.407284 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:16:46.422383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.422358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcsl\" (UniqueName: \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\") pod \"payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.462420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.462381 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msrn\" (UniqueName: \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.462622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.462606 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.563640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.563544 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msrn\" (UniqueName: \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.563826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.563718 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.564471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.564451 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.626229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.626199 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msrn\" (UniqueName: \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\") pod \"payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:46.699355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.699323 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:46.726662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:46.726632 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:47.086499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.086404 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:16:47.122351 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:47.122323 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb35a20a_2cd9_4349_a6d5_314422a061a4.slice/crio-60f5c3278f0c0a3f672ecd815f93c32f05ebecde3b79a17cf9ecccea21e3d8f4 WatchSource:0}: Error finding container 60f5c3278f0c0a3f672ecd815f93c32f05ebecde3b79a17cf9ecccea21e3d8f4: Status 404 returned error can't find the container with id 60f5c3278f0c0a3f672ecd815f93c32f05ebecde3b79a17cf9ecccea21e3d8f4 Nov 02 23:16:47.140592 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:16:47.140553 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c204dc_7776_4086_9a41_e57fbb4af357.slice/crio-c031f1f7897f6090fe5d0c697056c115c7e8b95e3267c15fd6b176a135ae322a WatchSource:0}: Error finding container c031f1f7897f6090fe5d0c697056c115c7e8b95e3267c15fd6b176a135ae322a: Status 404 returned error can't find the container with id c031f1f7897f6090fe5d0c697056c115c7e8b95e3267c15fd6b176a135ae322a Nov 02 23:16:47.141684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.141660 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:16:47.562578 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:16:47.562537 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:16:47.961332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.961294 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" event={"ID":"db35a20a-2cd9-4349-a6d5-314422a061a4","Type":"ContainerStarted","Data":"fd9b79bb26bb2a9ac35afee4b5c92fa8c68f88b9ca30de45ec0a6fd3f217d18c"} Nov 02 23:16:47.961778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.961360 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" event={"ID":"db35a20a-2cd9-4349-a6d5-314422a061a4","Type":"ContainerStarted","Data":"60f5c3278f0c0a3f672ecd815f93c32f05ebecde3b79a17cf9ecccea21e3d8f4"} Nov 02 23:16:47.961778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.961401 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:16:47.963398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.963373 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" event={"ID":"31c204dc-7776-4086-9a41-e57fbb4af357","Type":"ContainerStarted","Data":"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252"} Nov 02 23:16:47.963504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.963418 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" event={"ID":"31c204dc-7776-4086-9a41-e57fbb4af357","Type":"ContainerStarted","Data":"c031f1f7897f6090fe5d0c697056c115c7e8b95e3267c15fd6b176a135ae322a"} Nov 02 23:16:47.963504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:47.963449 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:48.054953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:48.054918 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:48.080217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:48.080140 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" podStartSLOduration=2.080121869 podStartE2EDuration="2.080121869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:48.077529896 +0000 UTC m=+1722.507399758" watchObservedRunningTime="2026-11-02 23:16:48.080121869 +0000 UTC m=+1722.509991744" Nov 02 23:16:53.043296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:53.043258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:53.046376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:53.046333 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" podStartSLOduration=7.046313998 podStartE2EDuration="7.046313998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:48.197368798 +0000 UTC m=+1722.627238659" watchObservedRunningTime="2026-11-02 23:16:53.046313998 +0000 UTC m=+1727.476183865" Nov 02 23:16:53.050475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:53.050452 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk"] Nov 02 23:16:53.119996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:53.119781 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c4c3385-c7slk"] Nov 02 23:16:53.694631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:53.694598 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edcd3d3-661d-4423-a151-75b803db33bf" path="/var/lib/kubelet/pods/9edcd3d3-661d-4423-a151-75b803db33bf/volumes" Nov 02 23:16:54.886473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:54.886440 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:16:54.886851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:54.886512 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:16:55.155996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:55.155888 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:55.311245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:55.311206 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:57.882840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:57.882805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:58.040775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:58.040733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:58.185602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:58.185551 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:16:58.973739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:58.973705 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:16:58.974306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:16:58.973770 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:17:02.471484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:02.471449 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:03.634494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:03.634448 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:05.208242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:05.208204 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:06.286511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:06.286454 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:07.099329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.099290 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:07.227075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.226985 4494 scope.go:122] "RemoveContainer" containerID="4640b9f2ef075b45e465e2dfba2c4a267b7024288730c381992002663955cb3c" Nov 02 23:17:07.247904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.247449 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:07.255108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.255085 4494 scope.go:122] "RemoveContainer" containerID="836fcd986525d47eeb6704881cfceadfbfb90242cb1bafc788aa396fb6aa911a" Nov 02 23:17:07.265720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.265697 4494 scope.go:122] "RemoveContainer" containerID="f4ec5fedaa84449423b0c74683198c2f06bf05e0521ce313cc9a5b1f283373ac" Nov 02 23:17:07.276598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.276569 4494 scope.go:122] "RemoveContainer" containerID="6a7ba5765d4baa9533ba1f4d6549fe744c2ea4996cbdb5beacfa6cddffe652cf" Nov 02 23:17:07.287484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.287466 4494 scope.go:122] "RemoveContainer" containerID="680da2f544d94ced14f752c7f875532a5dc3b4fcff428e5d4fd472b87fbfc064" Nov 02 23:17:07.297603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.297584 4494 scope.go:122] "RemoveContainer" containerID="0c968dc5f2808eac23e810417db50ebb77755b89e22bf64761b0ec4fe6760117" Nov 02 23:17:07.308231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.308211 4494 scope.go:122] "RemoveContainer" containerID="90ba2bae7585e6e43540e5e0762e9f203e7964aa23cda6a0201ce77e5c81a1dc" Nov 02 23:17:07.320391 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.320369 4494 scope.go:122] "RemoveContainer" containerID="7de8fae823b8725ce1a0736e96aa590ace505fd6b63dc7ea856b59a834fb9f72" Nov 02 23:17:07.746026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.745995 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:17:07.746271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.746225 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" containerID="cri-o://d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71" gracePeriod=30 Nov 02 23:17:07.758240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:07.758203 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:08.462855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:08.462818 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:08.609294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:08.609262 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:09.103879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:09.103835 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:17:09.104143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:09.104116 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" containerID="cri-o://53403f60e6d92263e0a0f3c35fde94aecd2a275375fe27f968837994f87ccb0e" gracePeriod=30 Nov 02 23:17:09.110116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:09.110091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:10.058279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.058240 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:10.586327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.586298 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:17:10.594004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.593983 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.656395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.656362 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b4bf70-openshift-default-dockercfg-2zwzn\"" Nov 02 23:17:10.717689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.717653 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750126 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750351 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750478 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750509 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750546 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9cj\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750574 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750612 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750652 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.751022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.750723 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.760200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.756424 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:17:10.890381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.890271 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.890756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.890733 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.891017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.890999 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.891152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.891136 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.891335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.891305 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9cj\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.891468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.891450 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.895196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.894282 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.895196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.894338 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.895196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.894420 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.898496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.898470 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.899384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.899350 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.899587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.899557 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.899858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.899839 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.900214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.900192 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.900731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.900703 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.904079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.903787 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.962835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.962802 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9cj\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:10.964601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:10.964575 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\") pod \"e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:11.118580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.118549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:11.239647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.239618 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:11.575068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.574986 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:11.708021 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:17:11.707395 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b02c19_bd94_436d_90b6_4a03f095856c.slice/crio-2eb1952c95fccb773d481b752c436879a526d090b234aaf6db4e59e2af5a47b6 WatchSource:0}: Error finding container 2eb1952c95fccb773d481b752c436879a526d090b234aaf6db4e59e2af5a47b6: Status 404 returned error can't find the container with id 2eb1952c95fccb773d481b752c436879a526d090b234aaf6db4e59e2af5a47b6 Nov 02 23:17:11.713496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.710973 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:17:11.713496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.711070 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:17:11.713496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.711115 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:17:11.720012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.719986 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:17:11.732228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.732197 4494 patch_prober.go:29] interesting pod/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:17:11.732329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:11.732253 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:17:12.277221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:12.277184 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerStarted","Data":"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd"} Nov 02 23:17:12.277683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:12.277243 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerStarted","Data":"2eb1952c95fccb773d481b752c436879a526d090b234aaf6db4e59e2af5a47b6"} Nov 02 23:17:12.402531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:12.402477 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podStartSLOduration=2.402464305 podStartE2EDuration="2.402464305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:12.398619642 +0000 UTC m=+1746.828489505" watchObservedRunningTime="2026-11-02 23:17:12.402464305 +0000 UTC m=+1746.832334167" Nov 02 23:17:13.007505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.007427 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:17:13.055614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055583 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.055614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055619 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.055861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055662 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.055861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055718 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.055861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055771 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.055861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055825 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.056109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055929 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.056109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.055999 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:13.056298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056105 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.056298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056105 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:13.056298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056125 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:13.056298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056208 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\") pod \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\" (UID: \"43399f3f-7d1a-468c-9037-a1e97fbed8fa\") " Nov 02 23:17:13.056632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056556 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:13.056749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056630 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:13.057073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056956 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.057073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.056986 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.057073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.057003 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.057073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.057017 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.057073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.057051 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.058094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.058073 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:13.058475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.058453 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:13.058567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.058526 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:13.058651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.058636 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc" pod "43399f3f-7d1a-468c-9037-a1e97fbed8fa" (UID: "43399f3f-7d1a-468c-9037-a1e97fbed8fa"). InnerVolumeSpecName "kube-api-access-tnblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:13.158108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.158077 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.158108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.158106 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnblc\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-kube-api-access-tnblc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.158108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.158116 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.158368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.158124 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43399f3f-7d1a-468c-9037-a1e97fbed8fa-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:13.240679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.240642 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:13.246769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.246740 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:13.286855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.286775 4494 generic.go:356] "Generic (PLEG): container finished" podID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerID="d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71" exitCode=0 Nov 02 23:17:13.286855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.286827 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" Nov 02 23:17:13.287331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.286855 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerDied","Data":"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71"} Nov 02 23:17:13.287331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.286914 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms" event={"ID":"43399f3f-7d1a-468c-9037-a1e97fbed8fa","Type":"ContainerDied","Data":"4380895973f71f337aa5961c1baddc43a0b32e22bcf609806914d8f92420e53b"} Nov 02 23:17:13.287331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.286945 4494 scope.go:122] "RemoveContainer" containerID="d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71" Nov 02 23:17:13.287331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.287285 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:13.288418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.288398 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:17:13.302315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.301904 4494 scope.go:122] "RemoveContainer" containerID="d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f" Nov 02 23:17:13.311670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.311651 4494 scope.go:122] "RemoveContainer" containerID="d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71" Nov 02 23:17:13.311942 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:17:13.311924 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71\": container with ID starting with d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71 not found: ID does not exist" containerID="d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71" Nov 02 23:17:13.312009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.311948 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71"} err="failed to get container status \"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71\": rpc error: code = NotFound desc = could not find container \"d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71\": container with ID starting with d0b971ac29c21b4e389de220f57f4c6e621ba1d71ecef5e17ad8aba47fef0e71 not found: ID does not exist" Nov 02 23:17:13.312009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.311965 4494 scope.go:122] "RemoveContainer" containerID="d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f" Nov 02 23:17:13.312258 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:17:13.312239 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f\": container with ID starting with d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f not found: ID does not exist" containerID="d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f" Nov 02 23:17:13.312323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.312262 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f"} err="failed to get container status \"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f\": rpc error: code = NotFound desc = could not find container \"d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f\": container with ID starting with d78019f071abfe0e1020c72f2e1357dc83f7596ff6b5fb6628dc7be6287eb36f not found: ID does not exist" Nov 02 23:17:13.485583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.485550 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:17:13.491245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.491221 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5153fe-openshift-default-5d6cc45b79-22jms"] Nov 02 23:17:13.695226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:13.695157 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" path="/var/lib/kubelet/pods/43399f3f-7d1a-468c-9037-a1e97fbed8fa/volumes" Nov 02 23:17:14.033212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.033107 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:17:14.145068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.145034 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:17:14.323474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.322204 4494 generic.go:356] "Generic (PLEG): container finished" podID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerID="53403f60e6d92263e0a0f3c35fde94aecd2a275375fe27f968837994f87ccb0e" exitCode=0 Nov 02 23:17:14.323474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.323255 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerDied","Data":"53403f60e6d92263e0a0f3c35fde94aecd2a275375fe27f968837994f87ccb0e"} Nov 02 23:17:14.323474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.323347 4494 scope.go:122] "RemoveContainer" containerID="2b5bb72e6da84bfd9a046bb0c506d404cd42f6dbca23b770245defda35771ae0" Nov 02 23:17:14.395214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.395164 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:14.405456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.405435 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:17:14.487865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.487822 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.488102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.488084 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.488377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.488356 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.488562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.488544 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.488794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.488777 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.488905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.488891 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.489101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.489075 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.489301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.489286 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.489435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.489419 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\") pod \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\" (UID: \"c9bfdfb4-0c5a-4063-ab7a-74885af48950\") " Nov 02 23:17:14.492057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.492032 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:14.493787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.493200 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:14.493787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.493704 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:14.493787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.493765 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:14.509296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.503321 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:14.575236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.570414 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:14.575236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.570513 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:14.575236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.574142 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:14.575236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.574405 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "kube-api-access-cbh4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:14.575236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.575020 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy" pod "c9bfdfb4-0c5a-4063-ab7a-74885af48950" (UID: "c9bfdfb4-0c5a-4063-ab7a-74885af48950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593748 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbh4s\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-kube-api-access-cbh4s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593783 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593801 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593814 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593830 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593844 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593857 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593879 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:14.593948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:14.593892 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bfdfb4-0c5a-4063-ab7a-74885af48950-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:15.197454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.197417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:15.330134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.330105 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" Nov 02 23:17:15.330648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.330131 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw" event={"ID":"c9bfdfb4-0c5a-4063-ab7a-74885af48950","Type":"ContainerDied","Data":"f46b8dc6181949ab98990774e425f6a5f7e2250e7a7682934380a5accef9f08b"} Nov 02 23:17:15.330648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.330228 4494 scope.go:122] "RemoveContainer" containerID="53403f60e6d92263e0a0f3c35fde94aecd2a275375fe27f968837994f87ccb0e" Nov 02 23:17:15.364863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.364825 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:15.501325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.501292 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:17:15.524099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.524054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:15.551428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.551402 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8ab111-openshift-default-6885bcb7bc-9smpw"] Nov 02 23:17:15.753147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.753064 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" path="/var/lib/kubelet/pods/c9bfdfb4-0c5a-4063-ab7a-74885af48950/volumes" Nov 02 23:17:15.932412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:15.932379 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:16.072354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:16.072236 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85"] Nov 02 23:17:16.120504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:16.120470 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ac1bb6-2zp85"] Nov 02 23:17:17.089379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:17.089346 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:17.564530 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:17:17.564448 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:17:17.695107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:17.695073 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdfde9a-92e0-4f94-ab3d-7683942e8b7d" path="/var/lib/kubelet/pods/afdfde9a-92e0-4f94-ab3d-7683942e8b7d/volumes" Nov 02 23:17:18.397250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.397218 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:18.629827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.629789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:18.822928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.822884 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:17:18.823897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823875 4494 cpu_manager.go:414] "containerMap: removing container" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:18.823984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823898 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:18.823984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823914 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:18.823984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823920 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:18.823984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823939 4494 cpu_manager.go:414] "containerMap: removing container" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:18.823984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.823945 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:18.824155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.824033 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:18.824155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.824043 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:18.831936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.831913 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:18.868144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.868107 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:18.868461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.868432 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jhj\" (UniqueName: \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:18.871269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.871247 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:17:18.926730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.926699 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4bf70-dockercfg-67b5f\"" Nov 02 23:17:18.956224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.950874 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4bf70\"" Nov 02 23:17:18.992697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.992656 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:18.993005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.992985 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jhj\" (UniqueName: \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:18.994336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.994308 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:19.000143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:18.999950 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:17:19.005004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.004982 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:19.005248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.005163 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:19.006191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.006149 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="43399f3f-7d1a-468c-9037-a1e97fbed8fa" containerName="istio-proxy" Nov 02 23:17:19.006360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.006347 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bfdfb4-0c5a-4063-ab7a-74885af48950" containerName="istio-proxy" Nov 02 23:17:19.067251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.041565 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" podUID="d9b92d4a-7076-4938-9911-d6baef3d917f" containerName="payload-processing" containerID="cri-o://511bbe99ad79f9c5a2084b1a32d8f167043c8541b609516d50cf3e1bbf6a5669" gracePeriod=25 Nov 02 23:17:19.137447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.137419 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:17:19.137631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.137550 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.138243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.138218 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jhj\" (UniqueName: \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:19.145986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.145959 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:19.146889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.146845 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" podUID="f8c505e8-3891-4fb9-8af8-35347d043298" containerName="payload-pre-processing" containerID="cri-o://72df31f4daeb249c989e23f28808d7e59b9f45335bc7012dcc4b6a815181526e" gracePeriod=25 Nov 02 23:17:19.213303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.213268 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq47j\" (UniqueName: \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.213445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.213327 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.314518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.314489 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq47j\" (UniqueName: \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.314677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.314528 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.315600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.315558 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.368129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.368034 4494 generic.go:356] "Generic (PLEG): container finished" podID="d9b92d4a-7076-4938-9911-d6baef3d917f" containerID="511bbe99ad79f9c5a2084b1a32d8f167043c8541b609516d50cf3e1bbf6a5669" exitCode=0 Nov 02 23:17:19.368129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.368107 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" event={"ID":"d9b92d4a-7076-4938-9911-d6baef3d917f","Type":"ContainerDied","Data":"511bbe99ad79f9c5a2084b1a32d8f167043c8541b609516d50cf3e1bbf6a5669"} Nov 02 23:17:19.370280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.370257 4494 generic.go:356] "Generic (PLEG): container finished" podID="f8c505e8-3891-4fb9-8af8-35347d043298" containerID="72df31f4daeb249c989e23f28808d7e59b9f45335bc7012dcc4b6a815181526e" exitCode=0 Nov 02 23:17:19.370411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.370311 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" event={"ID":"f8c505e8-3891-4fb9-8af8-35347d043298","Type":"ContainerDied","Data":"72df31f4daeb249c989e23f28808d7e59b9f45335bc7012dcc4b6a815181526e"} Nov 02 23:17:19.377914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.377889 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq47j\" (UniqueName: \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\") pod \"payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.413145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.413119 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:17:19.415514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.415487 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\" (UniqueName: \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\") pod \"d9b92d4a-7076-4938-9911-d6baef3d917f\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " Nov 02 23:17:19.415774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.415757 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\") pod \"d9b92d4a-7076-4938-9911-d6baef3d917f\" (UID: \"d9b92d4a-7076-4938-9911-d6baef3d917f\") " Nov 02 23:17:19.416546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.416520 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume" pod "d9b92d4a-7076-4938-9911-d6baef3d917f" (UID: "d9b92d4a-7076-4938-9911-d6baef3d917f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:19.420762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.420731 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn" pod "d9b92d4a-7076-4938-9911-d6baef3d917f" (UID: "d9b92d4a-7076-4938-9911-d6baef3d917f"). InnerVolumeSpecName "kube-api-access-jtjcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:19.446301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.446282 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:17:19.516586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.516548 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\" (UniqueName: \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\") pod \"f8c505e8-3891-4fb9-8af8-35347d043298\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " Nov 02 23:17:19.516746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.516636 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\") pod \"f8c505e8-3891-4fb9-8af8-35347d043298\" (UID: \"f8c505e8-3891-4fb9-8af8-35347d043298\") " Nov 02 23:17:19.517029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.517001 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume" pod "f8c505e8-3891-4fb9-8af8-35347d043298" (UID: "f8c505e8-3891-4fb9-8af8-35347d043298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:19.517146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.517126 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b92d4a-7076-4938-9911-d6baef3d917f-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:19.517229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.517145 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c505e8-3891-4fb9-8af8-35347d043298-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:19.517229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.517155 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjcn\" (UniqueName: \"kubernetes.io/projected/d9b92d4a-7076-4938-9911-d6baef3d917f-kube-api-access-jtjcn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:19.518824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.518806 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv" pod "f8c505e8-3891-4fb9-8af8-35347d043298" (UID: "f8c505e8-3891-4fb9-8af8-35347d043298"). InnerVolumeSpecName "kube-api-access-cx5tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:19.532006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.531952 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:17:19.532617 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:17:19.532593 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a705ba_336c_432e_87e7_1c5eaa463edc.slice/crio-321a66c5d865f352a8b0e99076819507f6ba75615ac826ececc50bc1d2ee71f1 WatchSource:0}: Error finding container 321a66c5d865f352a8b0e99076819507f6ba75615ac826ececc50bc1d2ee71f1: Status 404 returned error can't find the container with id 321a66c5d865f352a8b0e99076819507f6ba75615ac826ececc50bc1d2ee71f1 Nov 02 23:17:19.554465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.554432 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:19.618438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.618363 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx5tv\" (UniqueName: \"kubernetes.io/projected/f8c505e8-3891-4fb9-8af8-35347d043298-kube-api-access-cx5tv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:19.945986 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:17:19.945955 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62000536_2be8_4650_8e99_55ef15d9a081.slice/crio-182ea515c67429c80216a32a4fe0a59c36823ea7b7d4371a89ede94bb27706d3 WatchSource:0}: Error finding container 182ea515c67429c80216a32a4fe0a59c36823ea7b7d4371a89ede94bb27706d3: Status 404 returned error can't find the container with id 182ea515c67429c80216a32a4fe0a59c36823ea7b7d4371a89ede94bb27706d3 Nov 02 23:17:19.946401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:19.946377 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:17:20.378533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.378441 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" event={"ID":"42a705ba-336c-432e-87e7-1c5eaa463edc","Type":"ContainerStarted","Data":"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0"} Nov 02 23:17:20.378533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.378510 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:20.378735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.378537 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" event={"ID":"42a705ba-336c-432e-87e7-1c5eaa463edc","Type":"ContainerStarted","Data":"321a66c5d865f352a8b0e99076819507f6ba75615ac826ececc50bc1d2ee71f1"} Nov 02 23:17:20.379945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.379924 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" Nov 02 23:17:20.380069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.379964 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c" event={"ID":"d9b92d4a-7076-4938-9911-d6baef3d917f","Type":"ContainerDied","Data":"4aeb115318f23be6b1848e0cad8e349c242769de7e8dc6ce2e3fc2266582d93a"} Nov 02 23:17:20.380069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.380018 4494 scope.go:122] "RemoveContainer" containerID="511bbe99ad79f9c5a2084b1a32d8f167043c8541b609516d50cf3e1bbf6a5669" Nov 02 23:17:20.381875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.381843 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" event={"ID":"62000536-2be8-4650-8e99-55ef15d9a081","Type":"ContainerStarted","Data":"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803"} Nov 02 23:17:20.382059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.382040 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:20.382217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.382075 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" event={"ID":"62000536-2be8-4650-8e99-55ef15d9a081","Type":"ContainerStarted","Data":"182ea515c67429c80216a32a4fe0a59c36823ea7b7d4371a89ede94bb27706d3"} Nov 02 23:17:20.383723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.383705 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" Nov 02 23:17:20.383828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.383737 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2" event={"ID":"f8c505e8-3891-4fb9-8af8-35347d043298","Type":"ContainerDied","Data":"4e34f8484fc804d0f81bf7800a86cf5108412c2e7a9048a6d13ec2a59138184e"} Nov 02 23:17:20.390839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.390811 4494 scope.go:122] "RemoveContainer" containerID="72df31f4daeb249c989e23f28808d7e59b9f45335bc7012dcc4b6a815181526e" Nov 02 23:17:20.496723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.496664 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" podStartSLOduration=2.496646152 podStartE2EDuration="2.496646152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:20.495843818 +0000 UTC m=+1754.925713687" watchObservedRunningTime="2026-11-02 23:17:20.496646152 +0000 UTC m=+1754.926516015" Nov 02 23:17:20.739221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.739155 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" podStartSLOduration=2.739139537 podStartE2EDuration="2.739139537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:20.625239014 +0000 UTC m=+1755.055108876" watchObservedRunningTime="2026-11-02 23:17:20.739139537 +0000 UTC m=+1755.169009450" Nov 02 23:17:20.741001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.740982 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:17:20.746290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.746266 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-62ab55-76bbc656cf-l6q4c"] Nov 02 23:17:20.859076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.859040 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:17:20.862483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:20.862462 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-62ab55-6b5b4b5f7f-hr4q2"] Nov 02 23:17:21.695289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:21.695258 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b92d4a-7076-4938-9911-d6baef3d917f" path="/var/lib/kubelet/pods/d9b92d4a-7076-4938-9911-d6baef3d917f/volumes" Nov 02 23:17:21.695706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:21.695692 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c505e8-3891-4fb9-8af8-35347d043298" path="/var/lib/kubelet/pods/f8c505e8-3891-4fb9-8af8-35347d043298/volumes" Nov 02 23:17:21.748067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:21.748040 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:22.248696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:22.248660 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:24.189420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:24.189382 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:17:27.094224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:27.094188 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:27.250210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:27.250126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:27.447280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:27.447251 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:31.050366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:31.050335 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699"] Nov 02 23:17:31.100132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:31.100100 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf5db968-p9699"] Nov 02 23:17:31.395979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:31.395895 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:17:31.395979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:31.395972 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:17:31.695338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:31.695306 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a67d767-3ca7-4aad-86c7-88b2f8372e61" path="/var/lib/kubelet/pods/1a67d767-3ca7-4aad-86c7-88b2f8372e61/volumes" Nov 02 23:17:32.264818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:32.264786 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:37.098510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.098457 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:37.180052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.180008 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:17:37.180635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.180601 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" containerID="cri-o://0205842b73be582aa8f0c8aecfd9a603dfd57650e74ee21216a3fa7f534184e4" gracePeriod=30 Nov 02 23:17:37.241619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.241574 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:17:37.241883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.241844 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" containerID="cri-o://67ee3a5b4069c10a2b57b93634a24b1e947214adb0609d3d3f25a1ce480aa2cf" gracePeriod=30 Nov 02 23:17:37.245984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.245960 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:17:37.246366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.246337 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" containerID="cri-o://4a3133904d0f46278e080e806837121598cca1ab64b70371b9349789f4cba966" gracePeriod=30 Nov 02 23:17:37.267706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.267665 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:37.270233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.270206 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:17:37.270407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.270390 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" containerID="cri-o://50008c62fb4ac512cbd0dbb1b728831cf50d2a25778796e37f9d405172b8447f" gracePeriod=30 Nov 02 23:17:37.414567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:37.414539 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:40.093113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:40.093075 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:17:40.093604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:40.093151 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:17:41.335925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:41.335877 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:17:41.336326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:41.335963 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:17:42.258039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.258004 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:42.285828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.285797 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 23:17:42.285941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.285861 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 23:17:42.534100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.534069 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:42.540953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.540924 4494 generic.go:356] "Generic (PLEG): container finished" podID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerID="0205842b73be582aa8f0c8aecfd9a603dfd57650e74ee21216a3fa7f534184e4" exitCode=0 Nov 02 23:17:42.541123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.541073 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" event={"ID":"d1e06b5f-1966-4696-8a0c-b355d63692b1","Type":"ContainerDied","Data":"0205842b73be582aa8f0c8aecfd9a603dfd57650e74ee21216a3fa7f534184e4"} Nov 02 23:17:42.543260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.543233 4494 generic.go:356] "Generic (PLEG): container finished" podID="8f278669-ff51-4041-bc73-64448271df55" containerID="50008c62fb4ac512cbd0dbb1b728831cf50d2a25778796e37f9d405172b8447f" exitCode=0 Nov 02 23:17:42.543454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.543434 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" event={"ID":"8f278669-ff51-4041-bc73-64448271df55","Type":"ContainerDied","Data":"50008c62fb4ac512cbd0dbb1b728831cf50d2a25778796e37f9d405172b8447f"} Nov 02 23:17:42.545625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.545460 4494 generic.go:356] "Generic (PLEG): container finished" podID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerID="67ee3a5b4069c10a2b57b93634a24b1e947214adb0609d3d3f25a1ce480aa2cf" exitCode=0 Nov 02 23:17:42.545625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.545563 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" event={"ID":"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2","Type":"ContainerDied","Data":"67ee3a5b4069c10a2b57b93634a24b1e947214adb0609d3d3f25a1ce480aa2cf"} Nov 02 23:17:42.548024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.547550 4494 generic.go:356] "Generic (PLEG): container finished" podID="9e806f4d-804a-46b3-91ef-176211c3841d" containerID="4a3133904d0f46278e080e806837121598cca1ab64b70371b9349789f4cba966" exitCode=0 Nov 02 23:17:42.548024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.547765 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" event={"ID":"9e806f4d-804a-46b3-91ef-176211c3841d","Type":"ContainerDied","Data":"4a3133904d0f46278e080e806837121598cca1ab64b70371b9349789f4cba966"} Nov 02 23:17:42.575055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.575031 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:17:42.585234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.585163 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:17:42.622491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622453 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622510 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622547 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.622685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622575 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622696 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622726 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622804 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.622938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622856 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.622938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622895 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.623062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622941 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.622982 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623032 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623104 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623132 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.623266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623165 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\") pod \"d1e06b5f-1966-4696-8a0c-b355d63692b1\" (UID: \"d1e06b5f-1966-4696-8a0c-b355d63692b1\") " Nov 02 23:17:42.623266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623232 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623294 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.623491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623322 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\") pod \"9e806f4d-804a-46b3-91ef-176211c3841d\" (UID: \"9e806f4d-804a-46b3-91ef-176211c3841d\") " Nov 02 23:17:42.624365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.623827 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.624365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.624047 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.624600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.624520 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.624686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.624603 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.624883 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.624948 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.625080 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.625264 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.625483 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.625913 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626792 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626814 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626828 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e06b5f-1966-4696-8a0c-b355d63692b1-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626842 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626856 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626870 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626885 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e806f4d-804a-46b3-91ef-176211c3841d-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626898 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626911 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.626954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.626926 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.629813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.629763 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "kube-api-access-gvvjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.630118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.630017 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.630118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.630096 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:42.630419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.630391 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "kube-api-access-6pxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.630720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.630700 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.630992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.630970 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.631303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.631276 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo" pod "d1e06b5f-1966-4696-8a0c-b355d63692b1" (UID: "d1e06b5f-1966-4696-8a0c-b355d63692b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:42.632536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.632510 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy" pod "9e806f4d-804a-46b3-91ef-176211c3841d" (UID: "9e806f4d-804a-46b3-91ef-176211c3841d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.652965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.652943 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:17:42.665205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.665185 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:17:42.727596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727506 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.727596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727554 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727602 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727640 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727670 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727705 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727751 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.727817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727793 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727834 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727844 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727869 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.727945 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728008 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728042 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728083 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728089 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.728236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728156 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728248 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\") pod \"8f278669-ff51-4041-bc73-64448271df55\" (UID: \"8f278669-ff51-4041-bc73-64448271df55\") " Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728298 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728297 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728531 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728792 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.728842 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\") pod \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\" (UID: \"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2\") " Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729032 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729251 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729442 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729653 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729686 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729826 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729846 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729861 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f278669-ff51-4041-bc73-64448271df55-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729884 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729899 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729913 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e806f4d-804a-46b3-91ef-176211c3841d-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729926 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729941 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxqf\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-kube-api-access-6pxqf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729956 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e806f4d-804a-46b3-91ef-176211c3841d-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729969 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.729979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729983 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.729997 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730012 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e806f4d-804a-46b3-91ef-176211c3841d-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730026 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730041 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvjh\" (UniqueName: \"kubernetes.io/projected/d1e06b5f-1966-4696-8a0c-b355d63692b1-kube-api-access-gvvjh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730054 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.730867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730587 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.731223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730958 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:42.731223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.730966 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:42.731725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.731690 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:42.731916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.731898 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "kube-api-access-lvlmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.732277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.732256 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token" pod "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" (UID: "c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.732595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.732567 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "kube-api-access-grg69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.733266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.733243 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token" pod "8f278669-ff51-4041-bc73-64448271df55" (UID: "8f278669-ff51-4041-bc73-64448271df55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:42.831763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831728 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f278669-ff51-4041-bc73-64448271df55-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.831763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831757 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f278669-ff51-4041-bc73-64448271df55-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.831763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831767 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831777 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831786 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grg69\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-kube-api-access-grg69\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831794 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831802 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvlmk\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-kube-api-access-lvlmk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831813 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f278669-ff51-4041-bc73-64448271df55-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831823 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:42.832016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:42.831831 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:17:43.555916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.555836 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" Nov 02 23:17:43.555916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.555853 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm" event={"ID":"c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2","Type":"ContainerDied","Data":"76ba9b7b3993bd42a8232ea3354a0c732daf78b8dcf4e129246aeb92267b0cb6"} Nov 02 23:17:43.556441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.555916 4494 scope.go:122] "RemoveContainer" containerID="67ee3a5b4069c10a2b57b93634a24b1e947214adb0609d3d3f25a1ce480aa2cf" Nov 02 23:17:43.557360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.557345 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" Nov 02 23:17:43.557464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.557376 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9" event={"ID":"9e806f4d-804a-46b3-91ef-176211c3841d","Type":"ContainerDied","Data":"d556ae2f622c2e5e72d5c190c65e78c7574e1e0739b8b74141ad76d29c7db537"} Nov 02 23:17:43.559022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.558993 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" Nov 02 23:17:43.559120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.558999 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2" event={"ID":"d1e06b5f-1966-4696-8a0c-b355d63692b1","Type":"ContainerDied","Data":"20b336f8f7465d5815223fd991c651057dd437a34dbda1d2e7f50ec6a8098be0"} Nov 02 23:17:43.560654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.560635 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" event={"ID":"8f278669-ff51-4041-bc73-64448271df55","Type":"ContainerDied","Data":"1f84de758dff18384db4f20f3803a165f5641cb1eca75fefa7a8258bb4f02262"} Nov 02 23:17:43.560757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.560696 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9" Nov 02 23:17:43.574273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.574250 4494 scope.go:122] "RemoveContainer" containerID="4a3133904d0f46278e080e806837121598cca1ab64b70371b9349789f4cba966" Nov 02 23:17:43.585943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.585923 4494 scope.go:122] "RemoveContainer" containerID="0205842b73be582aa8f0c8aecfd9a603dfd57650e74ee21216a3fa7f534184e4" Nov 02 23:17:43.595417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.595397 4494 scope.go:122] "RemoveContainer" containerID="50008c62fb4ac512cbd0dbb1b728831cf50d2a25778796e37f9d405172b8447f" Nov 02 23:17:43.725374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.725339 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:17:43.775370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.775341 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-z42l2"] Nov 02 23:17:43.848239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.848100 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:17:43.848933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.848909 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-rxxl9"] Nov 02 23:17:43.954414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.954371 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:17:43.959464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:43.959439 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-r5rc9"] Nov 02 23:17:44.073526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:44.073481 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:17:44.081048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:44.081024 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-mscmm"] Nov 02 23:17:45.696717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:45.696674 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f278669-ff51-4041-bc73-64448271df55" path="/var/lib/kubelet/pods/8f278669-ff51-4041-bc73-64448271df55/volumes" Nov 02 23:17:45.697434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:45.697414 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" path="/var/lib/kubelet/pods/9e806f4d-804a-46b3-91ef-176211c3841d/volumes" Nov 02 23:17:45.698092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:45.698071 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" path="/var/lib/kubelet/pods/c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2/volumes" Nov 02 23:17:45.699195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:45.699158 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" path="/var/lib/kubelet/pods/d1e06b5f-1966-4696-8a0c-b355d63692b1/volumes" Nov 02 23:17:47.169157 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:47.169115 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:47.559936 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:17:47.559847 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:17:52.129284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:52.129248 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:17:55.059345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:55.059304 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc"] Nov 02 23:17:55.097865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:55.097832 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663c2edb-gwndc"] Nov 02 23:17:55.695843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:55.695799 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9bad71-702a-48b4-bafc-2cf309a166ac" path="/var/lib/kubelet/pods/1e9bad71-702a-48b4-bafc-2cf309a166ac/volumes" Nov 02 23:17:57.219035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:17:57.218995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:02.388587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:02.388547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:04.777451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.777416 4494 generic.go:356] "Generic (PLEG): container finished" podID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerID="2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f" exitCode=0 Nov 02 23:18:04.778033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.777455 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerDied","Data":"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f"} Nov 02 23:18:04.778033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.777919 4494 scope.go:122] "RemoveContainer" containerID="2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f" Nov 02 23:18:04.779453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779418 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:04.779592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779497 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:04.779592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779533 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:04.779592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779542 4494 generic.go:356] "Generic (PLEG): container finished" podID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerID="225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3" exitCode=0 Nov 02 23:18:04.779723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779613 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerDied","Data":"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3"} Nov 02 23:18:04.779982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.779963 4494 scope.go:122] "RemoveContainer" containerID="225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3" Nov 02 23:18:04.781226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.781196 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:04.781343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.781265 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:04.781343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:04.781308 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:05.285790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:05.285718 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:18:05.545255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:05.542735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:05.793375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:05.793335 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerStarted","Data":"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63"} Nov 02 23:18:05.797277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:05.796651 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerStarted","Data":"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd"} Nov 02 23:18:06.004825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.004788 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:18:06.006435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.006411 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:06.006507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.006464 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:06.380443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.380396 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:18:06.382147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.382103 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:06.382456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:06.382398 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:07.005750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.005716 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:07.006333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.006302 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:07.210342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.210306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:07.381828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.381728 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:07.382155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.382119 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:07.461300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.458269 4494 scope.go:122] "RemoveContainer" containerID="ac99c88afbaf0ddb36c61525702da02f74087a6dae0cc0826808ccb01d63a47a" Nov 02 23:18:07.586441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:07.584366 4494 scope.go:122] "RemoveContainer" containerID="77e772384a9df10e7b2c63a93126aa2a7d50894b24e5a1f659d229ddbf568781" Nov 02 23:18:08.004952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.004918 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:08.005124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.004967 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:08.389213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.383752 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:08.389213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.383884 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:08.916266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916229 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:18:08.916846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916832 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f8c505e8-3891-4fb9-8af8-35347d043298" containerName="payload-pre-processing" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916846 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c505e8-3891-4fb9-8af8-35347d043298" containerName="payload-pre-processing" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916862 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916867 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916887 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916892 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916905 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916909 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916918 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d9b92d4a-7076-4938-9911-d6baef3d917f" containerName="payload-processing" Nov 02 23:18:08.916925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916923 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b92d4a-7076-4938-9911-d6baef3d917f" containerName="payload-processing" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916933 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.916938 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917047 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c505e8-3891-4fb9-8af8-35347d043298" containerName="payload-pre-processing" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917055 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e806f4d-804a-46b3-91ef-176211c3841d" containerName="istio-proxy" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917065 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f278669-ff51-4041-bc73-64448271df55" containerName="istio-proxy" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917073 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a1b1f-85f2-4e8d-8e0f-0b606d6bced2" containerName="istio-proxy" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917080 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b92d4a-7076-4938-9911-d6baef3d917f" containerName="payload-processing" Nov 02 23:18:08.917285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.917086 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e06b5f-1966-4696-8a0c-b355d63692b1" containerName="istio-proxy" Nov 02 23:18:08.920623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:08.920601 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.005503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.005461 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:09.005681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.005641 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:09.006604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.006579 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbzh\" (UniqueName: \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.006699 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.006684 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.006792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.006779 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.006841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.006828 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.028256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.028224 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b4bf70-serving-cert\"" Nov 02 23:18:09.028399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.028331 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 23:18:09.028966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.028949 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 02 23:18:09.088203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.084875 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:18:09.108099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.108064 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.108346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.108269 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbzh\" (UniqueName: \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.108457 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:09.108367 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-serving-cert: secret "maas-api-e2e-ipp-a-b4bf70-serving-cert" not found Nov 02 23:18:09.108457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.108405 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.108457 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:09.108445 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls podName:b0286822-fd1a-4270-8d92-77e72cec367e nodeName:}" failed. No retries permitted until 2026-11-02 23:18:09.608420775 +0000 UTC m=+1804.038290635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls") pod "maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" (UID: "b0286822-fd1a-4270-8d92-77e72cec367e") : secret "maas-api-e2e-ipp-a-b4bf70-serving-cert" not found Nov 02 23:18:09.108647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.108513 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.109248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.109221 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.111587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.111565 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.117070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.117000 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:09.207260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.193682 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbzh\" (UniqueName: \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.274213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.266286 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:09.326514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.326484 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:18:09.389692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.389654 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:09.390430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.390395 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:09.467659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.467561 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:09.618928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.618896 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.621452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.621432 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:09.853230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:09.852412 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:10.006223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.005844 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:10.006223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.005927 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:10.291271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.291242 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:18:10.295104 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:10.295075 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0286822_fd1a_4270_8d92_77e72cec367e.slice/crio-23a63f4f273c642b711aa60f4b3d086612d103a9ccc1e33d40a3c3abf61b42c3 WatchSource:0}: Error finding container 23a63f4f273c642b711aa60f4b3d086612d103a9ccc1e33d40a3c3abf61b42c3: Status 404 returned error can't find the container with id 23a63f4f273c642b711aa60f4b3d086612d103a9ccc1e33d40a3c3abf61b42c3 Nov 02 23:18:10.296587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.296562 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:18:10.381199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.380592 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:10.381199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.380657 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:10.587773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.587631 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-2773f3"] Nov 02 23:18:10.592721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.592697 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:10.628728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.628658 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-2773f3"] Nov 02 23:18:10.656198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.635503 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz8h\" (UniqueName: \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\") pod \"test-curl-7705-2773f3\" (UID: \"72b24dc4-168b-4824-80ca-24c6aafe0851\") " pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:10.656198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.649058 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b8v4g\"" Nov 02 23:18:10.775201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.739068 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz8h\" (UniqueName: \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\") pod \"test-curl-7705-2773f3\" (UID: \"72b24dc4-168b-4824-80ca-24c6aafe0851\") " pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:10.806745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.806710 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz8h\" (UniqueName: \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\") pod \"test-curl-7705-2773f3\" (UID: \"72b24dc4-168b-4824-80ca-24c6aafe0851\") " pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:10.906735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.906627 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:10.918770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:10.918742 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" event={"ID":"b0286822-fd1a-4270-8d92-77e72cec367e","Type":"ContainerStarted","Data":"23a63f4f273c642b711aa60f4b3d086612d103a9ccc1e33d40a3c3abf61b42c3"} Nov 02 23:18:11.006234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.006196 4494 patch_prober.go:29] interesting pod/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 23:18:11.006538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.006260 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 23:18:11.340870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.340839 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-2773f3"] Nov 02 23:18:11.341294 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:11.341270 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72b24dc4_168b_4824_80ca_24c6aafe0851.slice/crio-d6c46cd2a9d891096bb50e282607726205e01a239dabff0d2622ba1340f7eaa4 WatchSource:0}: Error finding container d6c46cd2a9d891096bb50e282607726205e01a239dabff0d2622ba1340f7eaa4: Status 404 returned error can't find the container with id d6c46cd2a9d891096bb50e282607726205e01a239dabff0d2622ba1340f7eaa4 Nov 02 23:18:11.382196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.380660 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 23:18:11.382196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.380724 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 23:18:11.559774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.559735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:11.702757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.702725 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:11.928986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:11.928934 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-2773f3" event={"ID":"72b24dc4-168b-4824-80ca-24c6aafe0851","Type":"ContainerStarted","Data":"d6c46cd2a9d891096bb50e282607726205e01a239dabff0d2622ba1340f7eaa4"} Nov 02 23:18:12.010392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.010315 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:18:12.011627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.011605 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:18:12.030903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.030863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:12.183200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.182190 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:12.386668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.386595 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:18:12.387279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.387252 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:18:12.388000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.387978 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:18:12.942834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.942706 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" event={"ID":"b0286822-fd1a-4270-8d92-77e72cec367e","Type":"ContainerStarted","Data":"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748"} Nov 02 23:18:12.942834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:12.942788 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:13.085299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:13.083659 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" podStartSLOduration=2.909237591 podStartE2EDuration="5.08363518s" totalImagesPullingTime="2.174397589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:13.08170616 +0000 UTC m=+1807.511576028" watchObservedRunningTime="2026-11-02 23:18:13.08363518 +0000 UTC m=+1807.513505049" Nov 02 23:18:14.303865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.302354 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:14.609332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.609299 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:14.735219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.732942 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:18:14.735219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.733198 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" containerID="cri-o://e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d" gracePeriod=30 Nov 02 23:18:14.767152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.767123 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:14.982856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:14.982822 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-2773f3" event={"ID":"72b24dc4-168b-4824-80ca-24c6aafe0851","Type":"ContainerStarted","Data":"78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d"} Nov 02 23:18:15.099218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:15.099120 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7705-2773f3" podStartSLOduration=2.294948412 podStartE2EDuration="5.099102878s" totalImagesPullingTime="2.804154466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:15.097406111 +0000 UTC m=+1809.527275972" watchObservedRunningTime="2026-11-02 23:18:15.099102878 +0000 UTC m=+1809.528972740" Nov 02 23:18:15.935600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:15.935565 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:16.352848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.344624 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:16.450309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.450275 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:18:16.453373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.453354 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.509199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.508212 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d00785-openshift-default-dockercfg-glmf4\"" Nov 02 23:18:16.509199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.508739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510398 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510456 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510525 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s46s\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510553 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510604 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510632 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510658 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510693 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.534196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.510817 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.663308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.659603 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682544 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s46s\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682601 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682671 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682704 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682736 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682783 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.682951 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.683024 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.683074 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.684196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.684129 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.687048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.686957 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.687444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.687407 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.687746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.687728 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.688034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.688018 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.706305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.701770 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:18:16.708497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.708442 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-2773f3"] Nov 02 23:18:16.714981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.714917 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.719771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.719742 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.734318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.734267 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-2773f3"] Nov 02 23:18:16.746199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.746146 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.749923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.749854 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s46s\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\") pod \"e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.779780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.766530 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:16.974334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.974245 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:16.995939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:16.995897 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7705-2773f3" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" containerName="test-curl-7705-2773f3" containerID="cri-o://78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d" gracePeriod=2 Nov 02 23:18:17.047794 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:17.047759 4494 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7705-2773f3\" is forbidden: User \"system:node:ip-10-0-135-179.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-179.us-west-1.compute.internal' and this object" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:17.118613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:17.118579 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:17.228850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:17.228813 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:18:17.229697 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:17.229670 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4137d1d_8c99_463a_a5df_cdb4880a2e61.slice/crio-b672da4e93211ca673136fe7cf541e8a6b79c97a71b61dbe709c3fb63558a26d WatchSource:0}: Error finding container b672da4e93211ca673136fe7cf541e8a6b79c97a71b61dbe709c3fb63558a26d: Status 404 returned error can't find the container with id b672da4e93211ca673136fe7cf541e8a6b79c97a71b61dbe709c3fb63558a26d Nov 02 23:18:17.232362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:17.232323 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:17.232490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:17.232391 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:17.232490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:17.232426 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:17.563833 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:17.563728 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:18:18.004249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.004212 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerStarted","Data":"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18"} Nov 02 23:18:18.004685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.004274 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerStarted","Data":"b672da4e93211ca673136fe7cf541e8a6b79c97a71b61dbe709c3fb63558a26d"} Nov 02 23:18:18.128031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.127963 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podStartSLOduration=2.127945618 podStartE2EDuration="2.127945618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:18.125365877 +0000 UTC m=+1812.555235739" watchObservedRunningTime="2026-11-02 23:18:18.127945618 +0000 UTC m=+1812.557815479" Nov 02 23:18:18.780704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.780674 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:18.786413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.786385 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:18.957550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:18.957524 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:18:19.010638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.010617 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:19.011561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.011537 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:18:19.273434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.273406 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:19.321325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.321234 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\" (UniqueName: \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\") pod \"72b24dc4-168b-4824-80ca-24c6aafe0851\" (UID: \"72b24dc4-168b-4824-80ca-24c6aafe0851\") " Nov 02 23:18:19.323503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.323477 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h" pod "72b24dc4-168b-4824-80ca-24c6aafe0851" (UID: "72b24dc4-168b-4824-80ca-24c6aafe0851"). InnerVolumeSpecName "kube-api-access-wsz8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:19.422840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.422811 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsz8h\" (UniqueName: \"kubernetes.io/projected/72b24dc4-168b-4824-80ca-24c6aafe0851-kube-api-access-wsz8h\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:19.694541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.694507 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" path="/var/lib/kubelet/pods/72b24dc4-168b-4824-80ca-24c6aafe0851/volumes" Nov 02 23:18:19.978208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:19.978163 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:18:20.018479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.018448 4494 generic.go:356] "Generic (PLEG): container finished" podID="72b24dc4-168b-4824-80ca-24c6aafe0851" containerID="78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d" exitCode=137 Nov 02 23:18:20.018939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.018495 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-2773f3" Nov 02 23:18:20.018939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.018541 4494 scope.go:122] "RemoveContainer" containerID="78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d" Nov 02 23:18:20.020586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.020559 4494 generic.go:356] "Generic (PLEG): container finished" podID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerID="e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d" exitCode=0 Nov 02 23:18:20.020702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.020612 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" Nov 02 23:18:20.020778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.020741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerDied","Data":"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d"} Nov 02 23:18:20.020842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.020786 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw" event={"ID":"3d95a30d-fe5c-4f3b-81b7-e9b399516451","Type":"ContainerDied","Data":"c4a7e33146a07535496b31e4f4e27f90b4e3b8d5d1368ec899c23fea31d5ef0f"} Nov 02 23:18:20.028089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.028070 4494 scope.go:122] "RemoveContainer" containerID="78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d" Nov 02 23:18:20.028365 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:20.028344 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d\": container with ID starting with 78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d not found: ID does not exist" containerID="78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d" Nov 02 23:18:20.028442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.028371 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d"} err="failed to get container status \"78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d\": rpc error: code = NotFound desc = could not find container \"78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d\": container with ID starting with 78e23f829e8477b0a3cbf38ff8300e1393bce8ab7a37bd277759646fd8e3cc2d not found: ID does not exist" Nov 02 23:18:20.028442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.028391 4494 scope.go:122] "RemoveContainer" containerID="e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d" Nov 02 23:18:20.029346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029297 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.029437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029413 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:20.029511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029422 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.029574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029557 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.029622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029594 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.029962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029895 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.029962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029744 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.029962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029790 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.029962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.029960 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.030231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030060 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.030231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030086 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.030231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030134 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\") pod \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\" (UID: \"3d95a30d-fe5c-4f3b-81b7-e9b399516451\") " Nov 02 23:18:20.030407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030330 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.030482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030459 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.030675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030653 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.030795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030731 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.030795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030746 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.030795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030755 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.030795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030765 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.030795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.030778 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.032079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.032053 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.032206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.032098 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "kube-api-access-xlthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.032819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.032794 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:20.033427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.033410 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy" pod "3d95a30d-fe5c-4f3b-81b7-e9b399516451" (UID: "3d95a30d-fe5c-4f3b-81b7-e9b399516451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.056549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.056526 4494 scope.go:122] "RemoveContainer" containerID="56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.067915 4494 scope.go:122] "RemoveContainer" containerID="e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:20.068308 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d\": container with ID starting with e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d not found: ID does not exist" containerID="e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.068346 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d"} err="failed to get container status \"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d\": rpc error: code = NotFound desc = could not find container \"e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d\": container with ID starting with e3fd9c475c6c18d2d6bf0c77180364eddef9a882f360a548e966a11339db5a1d not found: ID does not exist" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.068369 4494 scope.go:122] "RemoveContainer" containerID="56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:20.068665 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10\": container with ID starting with 56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10 not found: ID does not exist" containerID="56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10" Nov 02 23:18:20.073387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.068692 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10"} err="failed to get container status \"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10\": rpc error: code = NotFound desc = could not find container \"56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10\": container with ID starting with 56c70a202bb10f8fcc947d116359f82c44d4083b769518fa63cc0a9773f49d10 not found: ID does not exist" Nov 02 23:18:20.132290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.132256 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.132290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.132286 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.132290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.132299 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d95a30d-fe5c-4f3b-81b7-e9b399516451-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.132522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.132311 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlthq\" (UniqueName: \"kubernetes.io/projected/3d95a30d-fe5c-4f3b-81b7-e9b399516451-kube-api-access-xlthq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.444068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.444026 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:18:20.447467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.447444 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-62ab55-openshift-default-74ff8b8697-gdtmw"] Nov 02 23:18:20.890483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:20.890454 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:21.079996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:21.079964 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:21.230427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:21.230388 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:21.694956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:21.694921 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" path="/var/lib/kubelet/pods/3d95a30d-fe5c-4f3b-81b7-e9b399516451/volumes" Nov 02 23:18:22.138577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:22.138493 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:24.106621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.106587 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:24.269199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.264092 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:18:24.319199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.318223 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:24.656040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656007 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:18:24.656600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656585 4494 cpu_manager.go:414] "containerMap: removing container" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" containerName="test-curl-7705-2773f3" Nov 02 23:18:24.656662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656602 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" containerName="test-curl-7705-2773f3" Nov 02 23:18:24.656662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656640 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.656662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656646 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.656766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656668 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.656766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656673 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.656766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656753 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.656766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.656765 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b24dc4-168b-4824-80ca-24c6aafe0851" containerName="test-curl-7705-2773f3" Nov 02 23:18:24.659921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.659902 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:24.717250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.717223 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:18:24.768459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.768426 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d00785\"" Nov 02 23:18:24.768837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.768812 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d00785-dockercfg-8hqfq\"" Nov 02 23:18:24.933236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.809824 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829xw\" (UniqueName: \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:24.933236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.928152 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:24.933236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.914798 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:18:24.933236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.930767 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d95a30d-fe5c-4f3b-81b7-e9b399516451" containerName="istio-proxy" Nov 02 23:18:24.933908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.933868 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:18:24.934037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:24.934015 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.032809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.032760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s44r\" (UniqueName: \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.033024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.032899 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.033024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.032991 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829xw\" (UniqueName: \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:25.033142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.033037 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:25.033760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.033739 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:25.095011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.094973 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829xw\" (UniqueName: \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\") pod \"payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:25.134625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.134588 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.135008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.134736 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s44r\" (UniqueName: \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.135326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.135307 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.196806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.196724 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s44r\" (UniqueName: \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\") pod \"payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.252008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.251976 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:25.271943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.271913 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:25.630660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.630567 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:18:25.670970 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:25.670929 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc88b847_04ff_446e_a440_a8584a69affb.slice/crio-384441406d8128347a7f2442e3e7ac64316238a688fa15c3479c72c9d72d0b1b WatchSource:0}: Error finding container 384441406d8128347a7f2442e3e7ac64316238a688fa15c3479c72c9d72d0b1b: Status 404 returned error can't find the container with id 384441406d8128347a7f2442e3e7ac64316238a688fa15c3479c72c9d72d0b1b Nov 02 23:18:25.708139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.708111 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:18:25.737468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:25.735835 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:26.081967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.081924 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" event={"ID":"d9ad77fd-737c-4fca-a18c-78a8469a8b05","Type":"ContainerStarted","Data":"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8"} Nov 02 23:18:26.082217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.081985 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" event={"ID":"d9ad77fd-737c-4fca-a18c-78a8469a8b05","Type":"ContainerStarted","Data":"bb8eeb23078b5fab9b379ad732cdfbd1680a66f33c86b887054aac2ac9989af8"} Nov 02 23:18:26.082217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.082018 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:26.083726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.083699 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" event={"ID":"dc88b847-04ff-446e-a440-a8584a69affb","Type":"ContainerStarted","Data":"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912"} Nov 02 23:18:26.083858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.083747 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" event={"ID":"dc88b847-04ff-446e-a440-a8584a69affb","Type":"ContainerStarted","Data":"384441406d8128347a7f2442e3e7ac64316238a688fa15c3479c72c9d72d0b1b"} Nov 02 23:18:26.083858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.083795 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:26.219946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.219895 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" podStartSLOduration=2.219881238 podStartE2EDuration="2.219881238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:26.218704157 +0000 UTC m=+1820.648574018" watchObservedRunningTime="2026-11-02 23:18:26.219881238 +0000 UTC m=+1820.649751099" Nov 02 23:18:26.370258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:26.364419 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" podStartSLOduration=2.364393696 podStartE2EDuration="2.364393696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:26.350157293 +0000 UTC m=+1820.780027155" watchObservedRunningTime="2026-11-02 23:18:26.364393696 +0000 UTC m=+1820.794263566" Nov 02 23:18:31.280789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:31.280752 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:34.668293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:34.668256 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:35.146803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.146709 4494 generic.go:356] "Generic (PLEG): container finished" podID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerID="4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd" exitCode=0 Nov 02 23:18:35.146803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.146780 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerDied","Data":"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd"} Nov 02 23:18:35.147251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.147236 4494 scope.go:122] "RemoveContainer" containerID="4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd" Nov 02 23:18:35.147404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.147385 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:18:35.148917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.148886 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:35.149016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.148950 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:35.149016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.148980 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:35.262782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.262724 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:18:35.509700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.509665 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:35.662529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:35.661868 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:36.149195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.147278 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:18:36.166256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.166216 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerStarted","Data":"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6"} Nov 02 23:18:36.166435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.166265 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.249201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.248496 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2b1c72-openshift-default-dockercfg-z888h\"" Nov 02 23:18:36.340207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.325509 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:18:36.340207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.326027 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:18:36.340207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.326129 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:36.340207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.326205 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:36.356305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.353124 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:36.560212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.531126 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.568779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.561098 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.568813 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.532636 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:36.569005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.568864 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.569150 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.569202 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.569234 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.569286 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczlq\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.569612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.569419 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.684255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.684142 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.688222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.684660 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.688485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.688458 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.697375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.697345 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.697670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.697647 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.697801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.697787 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.697912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.697899 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.698046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.698026 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczlq\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.698240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.698223 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.706200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.699565 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.706412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.692625 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.706850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.706826 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.707384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.707363 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.718200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.714234 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.728743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.727996 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.751196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.750785 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.754343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.754297 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.808200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.804927 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczlq\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\") pod \"e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:36.970198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.960482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:36.988199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:36.983052 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:37.109199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.108521 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:18:37.110745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.109965 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:18:37.263736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.261325 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:37.263736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.261382 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:37.560137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.559840 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:18:37.562025 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:37.561976 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d35d456_ed5a_45f2_b436_93d6d7d95369.slice/crio-468c08b326ea5f0aa85432ab28d71f53812e07a9df8a0d85cda8d6a30812110b WatchSource:0}: Error finding container 468c08b326ea5f0aa85432ab28d71f53812e07a9df8a0d85cda8d6a30812110b: Status 404 returned error can't find the container with id 468c08b326ea5f0aa85432ab28d71f53812e07a9df8a0d85cda8d6a30812110b Nov 02 23:18:37.565215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.565144 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:37.565335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.565243 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:37.565335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:37.565288 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:18:38.189404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.189360 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerStarted","Data":"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405"} Nov 02 23:18:38.189695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.189470 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerStarted","Data":"468c08b326ea5f0aa85432ab28d71f53812e07a9df8a0d85cda8d6a30812110b"} Nov 02 23:18:38.256324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.256289 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:38.256519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.256343 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:38.314656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.314605 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podStartSLOduration=2.314590936 podStartE2EDuration="2.314590936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:38.313246349 +0000 UTC m=+1832.743116210" watchObservedRunningTime="2026-11-02 23:18:38.314590936 +0000 UTC m=+1832.744460797" Nov 02 23:18:38.984505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.984474 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:38.989889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:38.989863 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:39.196238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:39.196207 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:39.197138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:39.197117 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:18:39.247679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:39.247584 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:39.247933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:39.247906 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:40.152217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.148259 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" podUID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" containerName="payload-processing" containerID="cri-o://997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c" gracePeriod=25 Nov 02 23:18:40.253194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.240566 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:40.253194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.240643 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:40.264277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.264235 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" podUID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" containerName="payload-pre-processing" containerID="cri-o://28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc" gracePeriod=25 Nov 02 23:18:40.525339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.525309 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:18:40.616039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.616009 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\" (UniqueName: \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\") pod \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " Nov 02 23:18:40.616262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.616244 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\") pod \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\" (UID: \"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f\") " Nov 02 23:18:40.616597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.616578 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume" pod "f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" (UID: "f67a3c46-eba0-4ee5-bf3c-c000c8c0339f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:40.616691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.616671 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:40.618202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.618147 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g" pod "f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" (UID: "f67a3c46-eba0-4ee5-bf3c-c000c8c0339f"). InnerVolumeSpecName "kube-api-access-r8d8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:40.668100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.668070 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:18:40.717777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.717746 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\" (UniqueName: \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\") pod \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " Nov 02 23:18:40.717975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.717810 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\") pod \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\" (UID: \"ecf2becf-9f2d-426d-b0fd-b618d1ce6850\") " Nov 02 23:18:40.718201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.718161 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume" pod "ecf2becf-9f2d-426d-b0fd-b618d1ce6850" (UID: "ecf2becf-9f2d-426d-b0fd-b618d1ce6850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:40.718277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.718237 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8d8g\" (UniqueName: \"kubernetes.io/projected/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f-kube-api-access-r8d8g\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:40.720041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.720022 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx" pod "ecf2becf-9f2d-426d-b0fd-b618d1ce6850" (UID: "ecf2becf-9f2d-426d-b0fd-b618d1ce6850"). InnerVolumeSpecName "kube-api-access-l7rsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:40.819964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.819878 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rsx\" (UniqueName: \"kubernetes.io/projected/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-kube-api-access-l7rsx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:40.819964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:40.819908 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf2becf-9f2d-426d-b0fd-b618d1ce6850-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:18:41.060689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.060655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:41.211438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.211403 4494 generic.go:356] "Generic (PLEG): container finished" podID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" containerID="28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc" exitCode=0 Nov 02 23:18:41.211894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.211475 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" Nov 02 23:18:41.211894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.211494 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" event={"ID":"ecf2becf-9f2d-426d-b0fd-b618d1ce6850","Type":"ContainerDied","Data":"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc"} Nov 02 23:18:41.211894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.211563 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6" event={"ID":"ecf2becf-9f2d-426d-b0fd-b618d1ce6850","Type":"ContainerDied","Data":"090c775d4dc8c75b6b89f12e8d48e95f9b117a04f1970d5b234b1c6bc558e41a"} Nov 02 23:18:41.211894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.211598 4494 scope.go:122] "RemoveContainer" containerID="28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc" Nov 02 23:18:41.213673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.213648 4494 generic.go:356] "Generic (PLEG): container finished" podID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" containerID="997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c" exitCode=0 Nov 02 23:18:41.213804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.213684 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" event={"ID":"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f","Type":"ContainerDied","Data":"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c"} Nov 02 23:18:41.213804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.213704 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" Nov 02 23:18:41.213804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.213737 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl" event={"ID":"f67a3c46-eba0-4ee5-bf3c-c000c8c0339f","Type":"ContainerDied","Data":"63a8fa9da5f348a28f8bfee5e46a964ff96a23bf37f1b0c5fad8a11f3085f1a6"} Nov 02 23:18:41.222928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.222913 4494 scope.go:122] "RemoveContainer" containerID="28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc" Nov 02 23:18:41.223280 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:41.223256 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc\": container with ID starting with 28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc not found: ID does not exist" containerID="28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc" Nov 02 23:18:41.223371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.223287 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc"} err="failed to get container status \"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc\": rpc error: code = NotFound desc = could not find container \"28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc\": container with ID starting with 28195346b41435a0a67a7515f40ca912ad69184e75af173e1c13e5cd8767e5bc not found: ID does not exist" Nov 02 23:18:41.223371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.223313 4494 scope.go:122] "RemoveContainer" containerID="997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c" Nov 02 23:18:41.247197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.245469 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:41.247727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.247701 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 02 23:18:41.247936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.247912 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 02 23:18:41.256519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.256501 4494 scope.go:122] "RemoveContainer" containerID="997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c" Nov 02 23:18:41.256936 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:41.256913 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c\": container with ID starting with 997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c not found: ID does not exist" containerID="997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c" Nov 02 23:18:41.257070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.256942 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c"} err="failed to get container status \"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c\": rpc error: code = NotFound desc = could not find container \"997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c\": container with ID starting with 997488caecf82479c9376de165befc0de3108477261ade3b89f28ab59594da6c not found: ID does not exist" Nov 02 23:18:41.402920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.402884 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:18:41.442020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.441926 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-76c8588567-x56gl"] Nov 02 23:18:41.471051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.471018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:41.551832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.551786 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:18:41.554756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.554735 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-6f95dc5cc9-sjfz6"] Nov 02 23:18:41.733272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.730638 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" path="/var/lib/kubelet/pods/ecf2becf-9f2d-426d-b0fd-b618d1ce6850/volumes" Nov 02 23:18:41.734221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:41.734195 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" path="/var/lib/kubelet/pods/f67a3c46-eba0-4ee5-bf3c-c000c8c0339f/volumes" Nov 02 23:18:42.244559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:42.244526 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:18:42.245315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:42.244984 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:18:42.245748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:42.245734 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:18:42.494650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:42.494560 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:44.063605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:44.063543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:45.637927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:45.637892 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:47.551520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:47.551434 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:47.562740 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:18:47.562701 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:18:47.697308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:47.697268 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:47.892796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:47.892684 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:48.098006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.097968 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:18:48.098848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.098818 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" containerName="payload-processing" Nov 02 23:18:48.098848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.098849 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" containerName="payload-processing" Nov 02 23:18:48.099029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.098933 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" containerName="payload-pre-processing" Nov 02 23:18:48.099029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.098941 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" containerName="payload-pre-processing" Nov 02 23:18:48.099127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.099081 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf2becf-9f2d-426d-b0fd-b618d1ce6850" containerName="payload-pre-processing" Nov 02 23:18:48.099127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.099103 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67a3c46-eba0-4ee5-bf3c-c000c8c0339f" containerName="payload-processing" Nov 02 23:18:48.104599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.104581 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.157800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.157706 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b1c72-dockercfg-cwtgv\"" Nov 02 23:18:48.158335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.158308 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b1c72\"" Nov 02 23:18:48.160996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.160967 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:18:48.263202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.262977 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bkw\" (UniqueName: \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.266425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.266387 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.290523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.290489 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:18:48.299955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.299534 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.335234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.330433 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:18:48.369302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.368782 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.369302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.368908 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bkw\" (UniqueName: \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.369302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.368998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbjm\" (UniqueName: \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.369302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.369095 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.369992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.369967 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.431495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.431468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bkw\" (UniqueName: \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:48.470472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.470426 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.470633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.470599 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbjm\" (UniqueName: \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.471202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.471159 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.532399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.532361 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbjm\" (UniqueName: \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\") pod \"payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.656161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.656125 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:48.718728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:48.718701 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:49.035515 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:49.035484 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f6f7b4_7f8f_4f40_9d0e_f203ac4e3820.slice/crio-972d353c33841158413dd3bf5d282deb225a6c33cd911e606b18aec87e7a32ef WatchSource:0}: Error finding container 972d353c33841158413dd3bf5d282deb225a6c33cd911e606b18aec87e7a32ef: Status 404 returned error can't find the container with id 972d353c33841158413dd3bf5d282deb225a6c33cd911e606b18aec87e7a32ef Nov 02 23:18:49.035859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.035832 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:18:49.134137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.134104 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:18:49.134459 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:18:49.134433 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5608db5a_403a_4263_89aa_c9e7c893bfd7.slice/crio-0e957653dc8a6d6afcbcba63955487313cd772e9a63d5bc12a4bb18a712d32be WatchSource:0}: Error finding container 0e957653dc8a6d6afcbcba63955487313cd772e9a63d5bc12a4bb18a712d32be: Status 404 returned error can't find the container with id 0e957653dc8a6d6afcbcba63955487313cd772e9a63d5bc12a4bb18a712d32be Nov 02 23:18:49.286575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.286494 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" event={"ID":"5608db5a-403a-4263-89aa-c9e7c893bfd7","Type":"ContainerStarted","Data":"56fc654610cc5d77f64a41b88cb9105380f88af9f324f384c40cc6e92f33b536"} Nov 02 23:18:49.286575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.286551 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" event={"ID":"5608db5a-403a-4263-89aa-c9e7c893bfd7","Type":"ContainerStarted","Data":"0e957653dc8a6d6afcbcba63955487313cd772e9a63d5bc12a4bb18a712d32be"} Nov 02 23:18:49.286784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.286675 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:18:49.288279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.288236 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" event={"ID":"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820","Type":"ContainerStarted","Data":"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a"} Nov 02 23:18:49.288279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.288290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" event={"ID":"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820","Type":"ContainerStarted","Data":"972d353c33841158413dd3bf5d282deb225a6c33cd911e606b18aec87e7a32ef"} Nov 02 23:18:49.288445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.288327 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:18:49.405533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.405480 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" podStartSLOduration=1.405466211 podStartE2EDuration="1.405466211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:49.402642093 +0000 UTC m=+1843.832511954" watchObservedRunningTime="2026-11-02 23:18:49.405466211 +0000 UTC m=+1843.835336071" Nov 02 23:18:49.536195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:49.535638 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" podStartSLOduration=1.535620433 podStartE2EDuration="1.535620433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:49.534696569 +0000 UTC m=+1843.964566430" watchObservedRunningTime="2026-11-02 23:18:49.535620433 +0000 UTC m=+1843.965490294" Nov 02 23:18:51.529792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:51.529755 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:52.663930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:52.663900 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:52.893400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:52.893365 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:53.040320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:53.040231 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:57.192819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:57.192785 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:57.938107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:57.938075 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:58.399997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:58.399902 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:18:58.516569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:58.516537 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:18:58.818091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:58.818053 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:58.996636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:58.995424 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:59.139048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:59.138967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:18:59.625703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:18:59.625668 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:00.298128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.298099 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:19:00.298357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.298164 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:19:00.919533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.919496 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:19:00.920040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.919728 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" containerID="cri-o://0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5" gracePeriod=30 Nov 02 23:19:00.924299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.924271 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:19:00.924503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.924481 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" containerID="cri-o://8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce" gracePeriod=30 Nov 02 23:19:00.972796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.972765 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 23:19:00.973319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:00.973289 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" containerID="cri-o://a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06" gracePeriod=30 Nov 02 23:19:02.444348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:02.444313 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:02.444741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:02.444366 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:03.192283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.192241 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:03.400774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.400728 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" podUID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" containerName="payload-processing" containerID="cri-o://c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868" gracePeriod=25 Nov 02 23:19:03.516959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.516921 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" podUID="e4349e3d-47d8-425b-a6dd-e752c947793c" containerName="payload-pre-processing" containerID="cri-o://abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075" gracePeriod=25 Nov 02 23:19:03.695574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.695542 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:19:03.718943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.718912 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\" (UniqueName: \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\") pod \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " Nov 02 23:19:03.719282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.719257 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\") pod \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\" (UID: \"0f10ed3f-b5dc-408a-9754-e20e4f14214e\") " Nov 02 23:19:03.719718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.719687 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume" pod "0f10ed3f-b5dc-408a-9754-e20e4f14214e" (UID: "0f10ed3f-b5dc-408a-9754-e20e4f14214e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:03.720089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.720063 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f10ed3f-b5dc-408a-9754-e20e4f14214e-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:03.722822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.722796 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5" pod "0f10ed3f-b5dc-408a-9754-e20e4f14214e" (UID: "0f10ed3f-b5dc-408a-9754-e20e4f14214e"). InnerVolumeSpecName "kube-api-access-r8bb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:03.796301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.796276 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:19:03.820567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.820535 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\") pod \"e4349e3d-47d8-425b-a6dd-e752c947793c\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " Nov 02 23:19:03.820748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.820678 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\" (UniqueName: \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\") pod \"e4349e3d-47d8-425b-a6dd-e752c947793c\" (UID: \"e4349e3d-47d8-425b-a6dd-e752c947793c\") " Nov 02 23:19:03.820971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.820944 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume" pod "e4349e3d-47d8-425b-a6dd-e752c947793c" (UID: "e4349e3d-47d8-425b-a6dd-e752c947793c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:03.821214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.821193 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4349e3d-47d8-425b-a6dd-e752c947793c-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:03.821322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.821218 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bb5\" (UniqueName: \"kubernetes.io/projected/0f10ed3f-b5dc-408a-9754-e20e4f14214e-kube-api-access-r8bb5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:03.822977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.822949 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2" pod "e4349e3d-47d8-425b-a6dd-e752c947793c" (UID: "e4349e3d-47d8-425b-a6dd-e752c947793c"). InnerVolumeSpecName "kube-api-access-bllc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:03.922156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:03.922122 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllc2\" (UniqueName: \"kubernetes.io/projected/e4349e3d-47d8-425b-a6dd-e752c947793c-kube-api-access-bllc2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:04.421908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.421872 4494 generic.go:356] "Generic (PLEG): container finished" podID="e4349e3d-47d8-425b-a6dd-e752c947793c" containerID="abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075" exitCode=0 Nov 02 23:19:04.422092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.421925 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" Nov 02 23:19:04.422092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.421953 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" event={"ID":"e4349e3d-47d8-425b-a6dd-e752c947793c","Type":"ContainerDied","Data":"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075"} Nov 02 23:19:04.422092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.422014 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9" event={"ID":"e4349e3d-47d8-425b-a6dd-e752c947793c","Type":"ContainerDied","Data":"d81e4ff1d986251a2e96f79ebc74391989aa7f1f8667d675e5ed776b9fbb4f02"} Nov 02 23:19:04.422092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.422042 4494 scope.go:122] "RemoveContainer" containerID="abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075" Nov 02 23:19:04.423822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.423626 4494 generic.go:356] "Generic (PLEG): container finished" podID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" containerID="c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868" exitCode=0 Nov 02 23:19:04.423896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.423827 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" event={"ID":"0f10ed3f-b5dc-408a-9754-e20e4f14214e","Type":"ContainerDied","Data":"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868"} Nov 02 23:19:04.423896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.423868 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" event={"ID":"0f10ed3f-b5dc-408a-9754-e20e4f14214e","Type":"ContainerDied","Data":"87d63545d03fc10938d0438e893f1fbe0d008d59d66568e2cb8939a165e6708d"} Nov 02 23:19:04.424013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.423930 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g" Nov 02 23:19:04.439468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.439446 4494 scope.go:122] "RemoveContainer" containerID="abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075" Nov 02 23:19:04.439813 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:04.439792 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075\": container with ID starting with abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075 not found: ID does not exist" containerID="abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075" Nov 02 23:19:04.439902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.439822 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075"} err="failed to get container status \"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075\": rpc error: code = NotFound desc = could not find container \"abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075\": container with ID starting with abf09eda57da175f6bf42b787f148db32313b98d42f09372901c3346fcafb075 not found: ID does not exist" Nov 02 23:19:04.439902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.439839 4494 scope.go:122] "RemoveContainer" containerID="c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868" Nov 02 23:19:04.448980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.448960 4494 scope.go:122] "RemoveContainer" containerID="c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868" Nov 02 23:19:04.449272 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:04.449253 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868\": container with ID starting with c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868 not found: ID does not exist" containerID="c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868" Nov 02 23:19:04.449326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.449280 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868"} err="failed to get container status \"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868\": rpc error: code = NotFound desc = could not find container \"c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868\": container with ID starting with c65582ac52efca34cc3cc4d0f974ddf93c517ce63ba1f609da808351edde0868 not found: ID does not exist" Nov 02 23:19:04.590870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.590836 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:19:04.639710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.639681 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7a67875-5949595c95-x6p7g"] Nov 02 23:19:04.751134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.751057 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:19:04.753841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:04.753821 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7a67875-6df9bb76cf-htbs9"] Nov 02 23:19:05.365964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.365910 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:05.572810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.572775 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q45zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:05.573008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.572842 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:05.694567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.694533 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" path="/var/lib/kubelet/pods/0f10ed3f-b5dc-408a-9754-e20e4f14214e/volumes" Nov 02 23:19:05.694940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.694918 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4349e3d-47d8-425b-a6dd-e752c947793c" path="/var/lib/kubelet/pods/e4349e3d-47d8-425b-a6dd-e752c947793c/volumes" Nov 02 23:19:05.880517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:05.880486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:06.259760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.259731 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:19:06.290813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.290789 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:19:06.294543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.294523 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 23:19:06.349564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349537 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.349732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349624 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.349732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349644 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.349732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349700 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349742 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349770 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349791 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349797 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349820 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349866 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.349901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349899 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349942 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.349990 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350016 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350046 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350072 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350087 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350057 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350097 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350204 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.350326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350275 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350362 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350370 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350473 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350556 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350569 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350593 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350651 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350692 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350698 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350736 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350774 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\") pod \"6965e205-f8fc-4266-b864-c5b3e16f5669\" (UID: \"6965e205-f8fc-4266-b864-c5b3e16f5669\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350811 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350846 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350917 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.350958 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.351028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351010 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\") pod \"23a4b29d-5679-456f-9aef-4e2a9082336f\" (UID: \"23a4b29d-5679-456f-9aef-4e2a9082336f\") " Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351040 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\") pod \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\" (UID: \"bfc44cd2-23ff-4641-98d6-0bf07dd75401\") " Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351441 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351698 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351722 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351720 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a4b29d-5679-456f-9aef-4e2a9082336f-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351795 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351869 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351907 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351921 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351931 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351934 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351968 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.351985 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6965e205-f8fc-4266-b864-c5b3e16f5669-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352000 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352016 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352029 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.352335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352141 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.353223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.353223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.352983 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:06.354147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.354118 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.354450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.354428 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "kube-api-access-cqpm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.354625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.354595 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.354760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.354735 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:06.355039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355014 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "kube-api-access-865s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.355108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355041 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token" pod "6965e205-f8fc-4266-b864-c5b3e16f5669" (UID: "6965e205-f8fc-4266-b864-c5b3e16f5669"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.355407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355381 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo" pod "bfc44cd2-23ff-4641-98d6-0bf07dd75401" (UID: "bfc44cd2-23ff-4641-98d6-0bf07dd75401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:06.355407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355396 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:06.355572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355517 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.355748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.355731 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4" pod "23a4b29d-5679-456f-9aef-4e2a9082336f" (UID: "23a4b29d-5679-456f-9aef-4e2a9082336f"). InnerVolumeSpecName "kube-api-access-rgrz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.442506 4494 generic.go:356] "Generic (PLEG): container finished" podID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerID="0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5" exitCode=0 Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.442549 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" event={"ID":"bfc44cd2-23ff-4641-98d6-0bf07dd75401","Type":"ContainerDied","Data":"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.442616 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" event={"ID":"bfc44cd2-23ff-4641-98d6-0bf07dd75401","Type":"ContainerDied","Data":"5dc8ea92c01431495d9244473313c50931448b69e9a9532774f68d80555ae231"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.442566 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.442647 4494 scope.go:122] "RemoveContainer" containerID="0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.444944 4494 generic.go:356] "Generic (PLEG): container finished" podID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerID="a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06" exitCode=0 Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.445069 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.445351 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" event={"ID":"23a4b29d-5679-456f-9aef-4e2a9082336f","Type":"ContainerDied","Data":"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.445399 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn" event={"ID":"23a4b29d-5679-456f-9aef-4e2a9082336f","Type":"ContainerDied","Data":"84897d1235ebb62185951d098d787bda01a7ba983ca3a27eb230132eb328bb63"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.446810 4494 generic.go:356] "Generic (PLEG): container finished" podID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerID="8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce" exitCode=0 Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.446856 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.446896 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" event={"ID":"6965e205-f8fc-4266-b864-c5b3e16f5669","Type":"ContainerDied","Data":"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.446937 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr" event={"ID":"6965e205-f8fc-4266-b864-c5b3e16f5669","Type":"ContainerDied","Data":"fc0c05d4f65f87838fd78e92e35a714e191db040da471b67ffa85bc98b973525"} Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452885 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452909 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452923 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpm5\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-kube-api-access-cqpm5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452936 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452949 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452961 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452974 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-865s4\" (UniqueName: \"kubernetes.io/projected/6965e205-f8fc-4266-b864-c5b3e16f5669-kube-api-access-865s4\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.452987 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrz4\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-kube-api-access-rgrz4\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453000 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453011 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453024 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a4b29d-5679-456f-9aef-4e2a9082336f-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453036 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a4b29d-5679-456f-9aef-4e2a9082336f-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453048 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc44cd2-23ff-4641-98d6-0bf07dd75401-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453060 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc44cd2-23ff-4641-98d6-0bf07dd75401-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.453073 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6965e205-f8fc-4266-b864-c5b3e16f5669-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.455674 4494 scope.go:122] "RemoveContainer" containerID="0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:06.455898 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5\": container with ID starting with 0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5 not found: ID does not exist" containerID="0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.455930 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5"} err="failed to get container status \"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5\": rpc error: code = NotFound desc = could not find container \"0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5\": container with ID starting with 0adf0b1ad07ee89f0d50cd146033108c22304550c912e49dad03421212c74fb5 not found: ID does not exist" Nov 02 23:19:06.459212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.455951 4494 scope.go:122] "RemoveContainer" containerID="a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06" Nov 02 23:19:06.467099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.467081 4494 scope.go:122] "RemoveContainer" containerID="a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06" Nov 02 23:19:06.469312 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:06.469290 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06\": container with ID starting with a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06 not found: ID does not exist" containerID="a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06" Nov 02 23:19:06.469424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.469317 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06"} err="failed to get container status \"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06\": rpc error: code = NotFound desc = could not find container \"a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06\": container with ID starting with a3e5ceb229381b6e1593c4dbd104be07265d2ba1b5fc60070b2ed3006f775b06 not found: ID does not exist" Nov 02 23:19:06.469424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.469334 4494 scope.go:122] "RemoveContainer" containerID="8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce" Nov 02 23:19:06.479086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.479064 4494 scope.go:122] "RemoveContainer" containerID="8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce" Nov 02 23:19:06.479494 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:06.479430 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce\": container with ID starting with 8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce not found: ID does not exist" containerID="8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce" Nov 02 23:19:06.479494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.479468 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce"} err="failed to get container status \"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce\": rpc error: code = NotFound desc = could not find container \"8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce\": container with ID starting with 8f85bf65fda2631fb271fa10137975cbd95bbd3c6c8d7ac465e5782d9cbe66ce not found: ID does not exist" Nov 02 23:19:06.793086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.793012 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:19:06.795812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.795789 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr"] Nov 02 23:19:06.904823 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.904786 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:19:06.908688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:06.908661 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q45zr"] Nov 02 23:19:07.019395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.019362 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 23:19:07.022751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.022731 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxqdn"] Nov 02 23:19:07.270116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.270075 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9qnvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:19:07.270335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.270142 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9qnvr" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:19:07.694116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.694080 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" path="/var/lib/kubelet/pods/23a4b29d-5679-456f-9aef-4e2a9082336f/volumes" Nov 02 23:19:07.694582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.694568 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" path="/var/lib/kubelet/pods/6965e205-f8fc-4266-b864-c5b3e16f5669/volumes" Nov 02 23:19:07.694970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.694958 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" path="/var/lib/kubelet/pods/bfc44cd2-23ff-4641-98d6-0bf07dd75401/volumes" Nov 02 23:19:07.994654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:07.994569 4494 scope.go:122] "RemoveContainer" containerID="c685352ea401b316167eeae411bd5fa35ba7ee4b2356447b5d967d3f63b102e7" Nov 02 23:19:08.004455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:08.004432 4494 scope.go:122] "RemoveContainer" containerID="16f9acc360a5685022ba52b0bbe59484d07ea44ede4be75e8c4b1c71f3dcf737" Nov 02 23:19:08.013636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:08.013610 4494 scope.go:122] "RemoveContainer" containerID="0f05f63008a220f74f574b69d6607d1abf9edbfc9f4259b6852de7d9984d8f82" Nov 02 23:19:08.347916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:08.347826 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:12.664962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:12.662441 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:14.121497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.121461 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:19:14.122057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122043 4494 cpu_manager.go:414] "containerMap: removing container" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" Nov 02 23:19:14.122109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122058 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" Nov 02 23:19:14.122109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122081 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e4349e3d-47d8-425b-a6dd-e752c947793c" containerName="payload-pre-processing" Nov 02 23:19:14.122109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122086 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4349e3d-47d8-425b-a6dd-e752c947793c" containerName="payload-pre-processing" Nov 02 23:19:14.122109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122100 4494 cpu_manager.go:414] "containerMap: removing container" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" Nov 02 23:19:14.122109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122104 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122112 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122116 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122123 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" containerName="payload-processing" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122127 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" containerName="payload-processing" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122264 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a4b29d-5679-456f-9aef-4e2a9082336f" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122275 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4349e3d-47d8-425b-a6dd-e752c947793c" containerName="payload-pre-processing" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122281 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc44cd2-23ff-4641-98d6-0bf07dd75401" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122288 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6965e205-f8fc-4266-b864-c5b3e16f5669" containerName="istio-proxy" Nov 02 23:19:14.122318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.122294 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f10ed3f-b5dc-408a-9754-e20e4f14214e" containerName="payload-processing" Nov 02 23:19:14.127499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.127481 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.184101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.184060 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:19:14.230911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.230880 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d00785-serving-cert\"" Nov 02 23:19:14.307756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.307718 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxpd\" (UniqueName: \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.307756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.307762 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.307971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.307865 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.307971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.307953 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.308931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.308901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:14.409311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.409289 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxpd\" (UniqueName: \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.409456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.409331 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.409456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.409388 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.409456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.409431 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.410316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.410288 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.411993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.411970 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.411993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.411982 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.502839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.502803 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxpd\" (UniqueName: \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\") pod \"maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:14.503554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.503524 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:14.645726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.645684 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:14.748639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:14.748556 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:15.127510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:15.127487 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:19:15.128133 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:15.128112 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d18244f_8a13_4338_b991_35e041a0790b.slice/crio-de53ae506f4dd164dc105130d55e97c7f0617de1cc980bde882fb13cba06d593 WatchSource:0}: Error finding container de53ae506f4dd164dc105130d55e97c7f0617de1cc980bde882fb13cba06d593: Status 404 returned error can't find the container with id de53ae506f4dd164dc105130d55e97c7f0617de1cc980bde882fb13cba06d593 Nov 02 23:19:15.495532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:15.495498 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:19:15.551957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:15.551927 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" event={"ID":"1d18244f-8a13-4338-b991-35e041a0790b","Type":"ContainerStarted","Data":"de53ae506f4dd164dc105130d55e97c7f0617de1cc980bde882fb13cba06d593"} Nov 02 23:19:16.560131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:16.560094 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" event={"ID":"1d18244f-8a13-4338-b991-35e041a0790b","Type":"ContainerStarted","Data":"c84c288a87f9be05c23d10273fe74984e0096ab2f953e6f8ec98a5feb22bda34"} Nov 02 23:19:16.560549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:16.560238 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:16.682659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:16.682598 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" podStartSLOduration=1.773107955 podStartE2EDuration="2.682583344s" totalImagesPullingTime="909.475389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:16.679380131 +0000 UTC m=+1871.109250002" watchObservedRunningTime="2026-11-02 23:19:16.682583344 +0000 UTC m=+1871.112453205" Nov 02 23:19:17.569208 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:17.566483 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:19:17.961642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:17.961606 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:18.836614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:18.836582 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:19.342924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.342893 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:19:19.347111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.347094 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.478200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.475073 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:19.497199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.493243 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:19:19.575880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.575849 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ee5b4-openshift-default-dockercfg-9qw2w\"" Nov 02 23:19:19.576451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.576426 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.576677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.576659 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.576825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.576812 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.577063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.577047 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.577238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.577224 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.577646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.577393 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.592196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.577869 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.604584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.604501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcc4\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.610234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.604816 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.632385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.631469 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716320 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716426 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716463 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcc4\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716528 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716617 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716642 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716681 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716787 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.716832 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.717339 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.717590 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.718709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.718445 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.719547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.719010 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.719691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.719650 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.720360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.720343 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.721058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.721032 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.780021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.779989 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcc4\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.780410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.780390 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\") pod \"e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:19.940158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.939886 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:19.961464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:19.961434 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:20.394548 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:20.394517 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4202e77_afe6_4be4_91ae_7de75abeaadb.slice/crio-8de68a4c2599dfd1d54c65e0efd455c1de0f9d0950fc4404cdca4272a82741b4 WatchSource:0}: Error finding container 8de68a4c2599dfd1d54c65e0efd455c1de0f9d0950fc4404cdca4272a82741b4: Status 404 returned error can't find the container with id 8de68a4c2599dfd1d54c65e0efd455c1de0f9d0950fc4404cdca4272a82741b4 Nov 02 23:19:20.395190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.395150 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:19:20.397113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.397082 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:20.397261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.397210 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:20.397338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.397262 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:20.625021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.624990 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerStarted","Data":"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123"} Nov 02 23:19:20.625278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.625039 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerStarted","Data":"8de68a4c2599dfd1d54c65e0efd455c1de0f9d0950fc4404cdca4272a82741b4"} Nov 02 23:19:20.748794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.748736 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podStartSLOduration=1.748718733 podStartE2EDuration="1.748718733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:20.746328242 +0000 UTC m=+1875.176198104" watchObservedRunningTime="2026-11-02 23:19:20.748718733 +0000 UTC m=+1875.178588593" Nov 02 23:19:20.962275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:20.962239 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:21.161573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:21.161543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:21.306420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:21.306385 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:21.942961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:21.942930 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:19:21.968371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:21.968343 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:22.052597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.052563 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:19:22.310896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.310798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:22.454186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.454138 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:22.575210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.575096 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:19:22.641235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.641200 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:22.642382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.642355 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:19:22.752585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:22.752549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:23.185286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:23.185251 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:19:23.185754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:23.185480 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" containerID="cri-o://6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7" gracePeriod=30 Nov 02 23:19:23.218518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:23.218479 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:24.020824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:24.020792 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:24.448853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:24.448819 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:24.663776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:24.663746 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:24.807896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:24.807818 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:25.026191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:25.026146 4494 patch_prober.go:29] interesting pod/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:19:25.026358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:25.026246 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:19:25.250872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:25.250838 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:26.943212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:26.943157 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" podUID="62000536-2be8-4650-8e99-55ef15d9a081" containerName="payload-processing" containerID="cri-o://9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803" gracePeriod=25 Nov 02 23:19:27.053086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.053051 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" podUID="42a705ba-336c-432e-87e7-1c5eaa463edc" containerName="payload-pre-processing" containerID="cri-o://f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0" gracePeriod=25 Nov 02 23:19:27.213249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.213191 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:19:27.323800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.323776 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:19:27.364417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.364383 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\") pod \"62000536-2be8-4650-8e99-55ef15d9a081\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " Nov 02 23:19:27.364574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.364478 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\" (UniqueName: \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\") pod \"62000536-2be8-4650-8e99-55ef15d9a081\" (UID: \"62000536-2be8-4650-8e99-55ef15d9a081\") " Nov 02 23:19:27.364861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.364832 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume" pod "62000536-2be8-4650-8e99-55ef15d9a081" (UID: "62000536-2be8-4650-8e99-55ef15d9a081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:27.366762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.366732 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j" pod "62000536-2be8-4650-8e99-55ef15d9a081" (UID: "62000536-2be8-4650-8e99-55ef15d9a081"). InnerVolumeSpecName "kube-api-access-zq47j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:27.465481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.465404 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\") pod \"42a705ba-336c-432e-87e7-1c5eaa463edc\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " Nov 02 23:19:27.465651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.465484 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\" (UniqueName: \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\") pod \"42a705ba-336c-432e-87e7-1c5eaa463edc\" (UID: \"42a705ba-336c-432e-87e7-1c5eaa463edc\") " Nov 02 23:19:27.465833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.465805 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume" pod "42a705ba-336c-432e-87e7-1c5eaa463edc" (UID: "42a705ba-336c-432e-87e7-1c5eaa463edc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:27.465985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.465954 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62000536-2be8-4650-8e99-55ef15d9a081-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:27.466070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.465989 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq47j\" (UniqueName: \"kubernetes.io/projected/62000536-2be8-4650-8e99-55ef15d9a081-kube-api-access-zq47j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:27.466070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.466003 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a705ba-336c-432e-87e7-1c5eaa463edc-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:27.467927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.467900 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj" pod "42a705ba-336c-432e-87e7-1c5eaa463edc" (UID: "42a705ba-336c-432e-87e7-1c5eaa463edc"). InnerVolumeSpecName "kube-api-access-77jhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:27.566958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.566928 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jhj\" (UniqueName: \"kubernetes.io/projected/42a705ba-336c-432e-87e7-1c5eaa463edc-kube-api-access-77jhj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:27.686623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.686582 4494 generic.go:356] "Generic (PLEG): container finished" podID="42a705ba-336c-432e-87e7-1c5eaa463edc" containerID="f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0" exitCode=0 Nov 02 23:19:27.686775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.686680 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" Nov 02 23:19:27.688294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.688265 4494 generic.go:356] "Generic (PLEG): container finished" podID="62000536-2be8-4650-8e99-55ef15d9a081" containerID="9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803" exitCode=0 Nov 02 23:19:27.688421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.688361 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" Nov 02 23:19:27.694911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.694883 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" event={"ID":"42a705ba-336c-432e-87e7-1c5eaa463edc","Type":"ContainerDied","Data":"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0"} Nov 02 23:19:27.695034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.694933 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8" event={"ID":"42a705ba-336c-432e-87e7-1c5eaa463edc","Type":"ContainerDied","Data":"321a66c5d865f352a8b0e99076819507f6ba75615ac826ececc50bc1d2ee71f1"} Nov 02 23:19:27.695034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.694958 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" event={"ID":"62000536-2be8-4650-8e99-55ef15d9a081","Type":"ContainerDied","Data":"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803"} Nov 02 23:19:27.695034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.694983 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs" event={"ID":"62000536-2be8-4650-8e99-55ef15d9a081","Type":"ContainerDied","Data":"182ea515c67429c80216a32a4fe0a59c36823ea7b7d4371a89ede94bb27706d3"} Nov 02 23:19:27.695034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.695005 4494 scope.go:122] "RemoveContainer" containerID="f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0" Nov 02 23:19:27.706885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.706869 4494 scope.go:122] "RemoveContainer" containerID="f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0" Nov 02 23:19:27.707939 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:27.707919 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0\": container with ID starting with f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0 not found: ID does not exist" containerID="f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0" Nov 02 23:19:27.708023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.707946 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0"} err="failed to get container status \"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0\": rpc error: code = NotFound desc = could not find container \"f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0\": container with ID starting with f93f2e0b4780f26199d10d4497d735fe1405df3571e52610eece696fac88cae0 not found: ID does not exist" Nov 02 23:19:27.708023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.707964 4494 scope.go:122] "RemoveContainer" containerID="9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803" Nov 02 23:19:27.717425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.717407 4494 scope.go:122] "RemoveContainer" containerID="9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803" Nov 02 23:19:27.717714 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:27.717691 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803\": container with ID starting with 9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803 not found: ID does not exist" containerID="9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803" Nov 02 23:19:27.717812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.717718 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803"} err="failed to get container status \"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803\": rpc error: code = NotFound desc = could not find container \"9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803\": container with ID starting with 9b33ce94ccd503c60c3ddd2897be91abeef1a762c6450d28208b5fd3ad210803 not found: ID does not exist" Nov 02 23:19:27.853816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.853774 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:19:27.904455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.904428 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-685cb86875-lb5w8"] Nov 02 23:19:27.969737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.969666 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:19:27.972403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:27.972379 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-76f866898c-pgdbs"] Nov 02 23:19:28.249734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.249674 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:28.456130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.456098 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:28.645243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.645126 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:19:28.661552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.661526 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:19:28.662108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662094 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.662155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662109 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.662225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662157 4494 cpu_manager.go:414] "containerMap: removing container" podUID="62000536-2be8-4650-8e99-55ef15d9a081" containerName="payload-processing" Nov 02 23:19:28.662225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662163 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62000536-2be8-4650-8e99-55ef15d9a081" containerName="payload-processing" Nov 02 23:19:28.662225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662187 4494 cpu_manager.go:414] "containerMap: removing container" podUID="42a705ba-336c-432e-87e7-1c5eaa463edc" containerName="payload-pre-processing" Nov 02 23:19:28.662225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662194 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a705ba-336c-432e-87e7-1c5eaa463edc" containerName="payload-pre-processing" Nov 02 23:19:28.662361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662297 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.662361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662309 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.662361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662317 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="62000536-2be8-4650-8e99-55ef15d9a081" containerName="payload-processing" Nov 02 23:19:28.662361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.662325 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a705ba-336c-432e-87e7-1c5eaa463edc" containerName="payload-pre-processing" Nov 02 23:19:28.665545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.665530 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.697980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.697931 4494 generic.go:356] "Generic (PLEG): container finished" podID="3407dc7f-387d-4f50-950b-fa841a010942" containerID="6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7" exitCode=0 Nov 02 23:19:28.698210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.697983 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerDied","Data":"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7"} Nov 02 23:19:28.698210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.697991 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" Nov 02 23:19:28.698210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.698040 4494 scope.go:122] "RemoveContainer" containerID="6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7" Nov 02 23:19:28.698210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.698029 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q" event={"ID":"3407dc7f-387d-4f50-950b-fa841a010942","Type":"ContainerDied","Data":"f3a812b29e611f4839eb2b1d415dab16c459e1da4439ad6fb5b0775904d5871a"} Nov 02 23:19:28.707917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.707896 4494 scope.go:122] "RemoveContainer" containerID="2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3" Nov 02 23:19:28.716985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.716953 4494 scope.go:122] "RemoveContainer" containerID="6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7" Nov 02 23:19:28.717273 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:28.717251 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7\": container with ID starting with 6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7 not found: ID does not exist" containerID="6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7" Nov 02 23:19:28.717360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.717278 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7"} err="failed to get container status \"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7\": rpc error: code = NotFound desc = could not find container \"6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7\": container with ID starting with 6d72c97c87d8fbfc9eecc8d8288ff9a1e43adf4da6eda3fe36858e71a64a31a7 not found: ID does not exist" Nov 02 23:19:28.717360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.717296 4494 scope.go:122] "RemoveContainer" containerID="2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3" Nov 02 23:19:28.717546 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:28.717526 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3\": container with ID starting with 2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3 not found: ID does not exist" containerID="2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3" Nov 02 23:19:28.717594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.717554 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3"} err="failed to get container status \"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3\": rpc error: code = NotFound desc = could not find container \"2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3\": container with ID starting with 2542bbb6898c5db5e4c61001bc30a13ba511646f23da47aea7270fcb0b1aa5d3 not found: ID does not exist" Nov 02 23:19:28.719726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.719363 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ee5b4\"" Nov 02 23:19:28.719726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.719555 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ee5b4-dockercfg-jqk99\"" Nov 02 23:19:28.723090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.723067 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.797782 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.797880 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.797926 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.797989 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.798051 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.798157 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.798267 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.798308 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.801196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.798370 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\") pod \"3407dc7f-387d-4f50-950b-fa841a010942\" (UID: \"3407dc7f-387d-4f50-950b-fa841a010942\") " Nov 02 23:19:28.828671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.828636 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:28.829522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.829435 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:28.831029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.830995 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k4gw\" (UniqueName: \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.831201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.831164 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.831383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.831367 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.831452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.831391 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3407dc7f-387d-4f50-950b-fa841a010942-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.833369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.833091 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:28.834485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.834459 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:28.834714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.834693 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:28.851034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.841254 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:28.855751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.855699 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "kube-api-access-qcptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:28.856628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.856600 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:28.858533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.858495 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy" pod "3407dc7f-387d-4f50-950b-fa841a010942" (UID: "3407dc7f-387d-4f50-950b-fa841a010942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:28.862642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.862390 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:19:28.866193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.863448 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.866193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.863470 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3407dc7f-387d-4f50-950b-fa841a010942" containerName="istio-proxy" Nov 02 23:19:28.871324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.871297 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:28.888237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.886885 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:19:28.931998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.931958 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.932199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932139 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k4gw\" (UniqueName: \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.932260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932246 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932263 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcptj\" (UniqueName: \"kubernetes.io/projected/3407dc7f-387d-4f50-950b-fa841a010942-kube-api-access-qcptj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932277 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932289 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932303 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3407dc7f-387d-4f50-950b-fa841a010942-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932314 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932327 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3407dc7f-387d-4f50-950b-fa841a010942-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:28.932839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.932817 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:28.994372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:28.994343 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k4gw\" (UniqueName: \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\") pod \"payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:29.033029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.032996 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.033225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.033203 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslkl\" (UniqueName: \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.070902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.070870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:29.149356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.145494 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.149356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.145632 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslkl\" (UniqueName: \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.149356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.147361 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.207652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.207624 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslkl\" (UniqueName: \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\") pod \"payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.224654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.224621 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:19:29.228830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.228808 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bf51d6-openshift-default-5874f88457-tv84q"] Nov 02 23:19:29.287197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.281426 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:29.400743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.400509 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:19:29.493606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.493577 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:29.554609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.554568 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:29.564893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.564863 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:19:29.694738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.694506 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3407dc7f-387d-4f50-950b-fa841a010942" path="/var/lib/kubelet/pods/3407dc7f-387d-4f50-950b-fa841a010942/volumes" Nov 02 23:19:29.695199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.695073 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a705ba-336c-432e-87e7-1c5eaa463edc" path="/var/lib/kubelet/pods/42a705ba-336c-432e-87e7-1c5eaa463edc/volumes" Nov 02 23:19:29.695667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.695652 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62000536-2be8-4650-8e99-55ef15d9a081" path="/var/lib/kubelet/pods/62000536-2be8-4650-8e99-55ef15d9a081/volumes" Nov 02 23:19:29.696350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.696330 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:19:29.697747 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:29.697724 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f671d2_fea5_4493_89c6_6042de61b05d.slice/crio-e16b70a081760229488cc7a6aec9e51e5826e506679146ed07fc5044616a74cc WatchSource:0}: Error finding container e16b70a081760229488cc7a6aec9e51e5826e506679146ed07fc5044616a74cc: Status 404 returned error can't find the container with id e16b70a081760229488cc7a6aec9e51e5826e506679146ed07fc5044616a74cc Nov 02 23:19:29.706061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.706042 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" event={"ID":"d0f671d2-fea5-4493-89c6-6042de61b05d","Type":"ContainerStarted","Data":"e16b70a081760229488cc7a6aec9e51e5826e506679146ed07fc5044616a74cc"} Nov 02 23:19:29.889342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.888248 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:29.974319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:29.974294 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:19:29.975751 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:29.975722 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6588ecac_78f7_49c3_941d_d6fa831b386b.slice/crio-24286e7fcdc86e806e8c4c6ab434d1442e2120394d307d35e3325095b7c74ef8 WatchSource:0}: Error finding container 24286e7fcdc86e806e8c4c6ab434d1442e2120394d307d35e3325095b7c74ef8: Status 404 returned error can't find the container with id 24286e7fcdc86e806e8c4c6ab434d1442e2120394d307d35e3325095b7c74ef8 Nov 02 23:19:30.047411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.043517 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:30.649492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.649460 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:19:30.654729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.654706 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.706697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.706664 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:19:30.708377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.708351 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-aa2454\"" Nov 02 23:19:30.708547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.708430 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-aa2454-dockercfg-brm8f\"" Nov 02 23:19:30.718000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.717970 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" event={"ID":"6588ecac-78f7-49c3-941d-d6fa831b386b","Type":"ContainerStarted","Data":"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca"} Nov 02 23:19:30.718186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.718035 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:30.718186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.718059 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" event={"ID":"6588ecac-78f7-49c3-941d-d6fa831b386b","Type":"ContainerStarted","Data":"24286e7fcdc86e806e8c4c6ab434d1442e2120394d307d35e3325095b7c74ef8"} Nov 02 23:19:30.720082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.720055 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" event={"ID":"d0f671d2-fea5-4493-89c6-6042de61b05d","Type":"ContainerStarted","Data":"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e"} Nov 02 23:19:30.720239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.720138 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:30.804401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.804374 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhjj\" (UniqueName: \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.804550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.804521 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.887880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.887829 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" podStartSLOduration=2.88781567 podStartE2EDuration="2.88781567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:30.886476588 +0000 UTC m=+1885.316346448" watchObservedRunningTime="2026-11-02 23:19:30.88781567 +0000 UTC m=+1885.317685528" Nov 02 23:19:30.906046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.905958 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhjj\" (UniqueName: \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.906228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.906091 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.906842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.906819 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.967894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.967868 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhjj\" (UniqueName: \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\") pod \"payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:30.968738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:30.968719 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:31.020211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.019539 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" podStartSLOduration=3.019520288 podStartE2EDuration="3.019520288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:31.018692312 +0000 UTC m=+1885.448562173" watchObservedRunningTime="2026-11-02 23:19:31.019520288 +0000 UTC m=+1885.449390151" Nov 02 23:19:31.181097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.180850 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:31.375123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.375094 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:19:31.380274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.380252 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.437214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.437109 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:19:31.453632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.453601 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.453796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.453670 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cx6\" (UniqueName: \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.487455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.487430 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:19:31.487838 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:31.487818 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880f056c_108b_4abc_a5a7_ac1c1bb98dd7.slice/crio-d2077f35e7d87e7de47fb5fb4c914f59a3a5fe34c7c18c8ae4202b5e912145cf WatchSource:0}: Error finding container d2077f35e7d87e7de47fb5fb4c914f59a3a5fe34c7c18c8ae4202b5e912145cf: Status 404 returned error can't find the container with id d2077f35e7d87e7de47fb5fb4c914f59a3a5fe34c7c18c8ae4202b5e912145cf Nov 02 23:19:31.554486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.554447 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.554669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.554533 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cx6\" (UniqueName: \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.555113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.555094 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.616135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.616104 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cx6\" (UniqueName: \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\") pod \"payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.694623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.694551 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:31.729463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:31.729432 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" event={"ID":"880f056c-108b-4abc-a5a7-ac1c1bb98dd7","Type":"ContainerStarted","Data":"d2077f35e7d87e7de47fb5fb4c914f59a3a5fe34c7c18c8ae4202b5e912145cf"} Nov 02 23:19:32.071060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:32.071032 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:19:32.071627 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:32.071601 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e97d3_63b8_46b5_84cf_9be382585fcc.slice/crio-63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7 WatchSource:0}: Error finding container 63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7: Status 404 returned error can't find the container with id 63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7 Nov 02 23:19:32.576086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:32.576052 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:32.739257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:32.739214 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" event={"ID":"1d0e97d3-63b8-46b5-84cf-9be382585fcc","Type":"ContainerStarted","Data":"63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7"} Nov 02 23:19:34.401166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.401119 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" podUID="31c204dc-7776-4086-9a41-e57fbb4af357" containerName="payload-processing" containerID="cri-o://ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252" gracePeriod=25 Nov 02 23:19:34.565926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.565884 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" podUID="db35a20a-2cd9-4349-a6d5-314422a061a4" containerName="payload-pre-processing" containerID="cri-o://fd9b79bb26bb2a9ac35afee4b5c92fa8c68f88b9ca30de45ec0a6fd3f217d18c" gracePeriod=25 Nov 02 23:19:34.679375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.679347 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:19:34.700029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.699973 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\" (UniqueName: \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\") pod \"31c204dc-7776-4086-9a41-e57fbb4af357\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " Nov 02 23:19:34.700227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.700055 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\") pod \"31c204dc-7776-4086-9a41-e57fbb4af357\" (UID: \"31c204dc-7776-4086-9a41-e57fbb4af357\") " Nov 02 23:19:34.701022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.700459 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume" pod "31c204dc-7776-4086-9a41-e57fbb4af357" (UID: "31c204dc-7776-4086-9a41-e57fbb4af357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:34.701022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.700830 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c204dc-7776-4086-9a41-e57fbb4af357-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:34.702348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.702324 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn" pod "31c204dc-7776-4086-9a41-e57fbb4af357" (UID: "31c204dc-7776-4086-9a41-e57fbb4af357"). InnerVolumeSpecName "kube-api-access-9msrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:34.757820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.757790 4494 generic.go:356] "Generic (PLEG): container finished" podID="db35a20a-2cd9-4349-a6d5-314422a061a4" containerID="fd9b79bb26bb2a9ac35afee4b5c92fa8c68f88b9ca30de45ec0a6fd3f217d18c" exitCode=0 Nov 02 23:19:34.758006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.757866 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" event={"ID":"db35a20a-2cd9-4349-a6d5-314422a061a4","Type":"ContainerDied","Data":"fd9b79bb26bb2a9ac35afee4b5c92fa8c68f88b9ca30de45ec0a6fd3f217d18c"} Nov 02 23:19:34.759273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.759251 4494 generic.go:356] "Generic (PLEG): container finished" podID="31c204dc-7776-4086-9a41-e57fbb4af357" containerID="ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252" exitCode=0 Nov 02 23:19:34.759397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.759305 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" Nov 02 23:19:34.759397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.759336 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" event={"ID":"31c204dc-7776-4086-9a41-e57fbb4af357","Type":"ContainerDied","Data":"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252"} Nov 02 23:19:34.759397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.759393 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n" event={"ID":"31c204dc-7776-4086-9a41-e57fbb4af357","Type":"ContainerDied","Data":"c031f1f7897f6090fe5d0c697056c115c7e8b95e3267c15fd6b176a135ae322a"} Nov 02 23:19:34.759609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.759424 4494 scope.go:122] "RemoveContainer" containerID="ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252" Nov 02 23:19:34.764244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.761297 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" event={"ID":"1d0e97d3-63b8-46b5-84cf-9be382585fcc","Type":"ContainerStarted","Data":"c8adb2d62d9e92f9d77a691687360c12f45fa6330c65ac6e66978d7fae6ba907"} Nov 02 23:19:34.764244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.761379 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:34.764244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.763716 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" event={"ID":"880f056c-108b-4abc-a5a7-ac1c1bb98dd7","Type":"ContainerStarted","Data":"5f42f40ccabd03dc8f730e234816139364687046ddef07bd43230025c15e62c2"} Nov 02 23:19:34.764244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.763780 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:34.775615 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.775599 4494 scope.go:122] "RemoveContainer" containerID="ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252" Nov 02 23:19:34.775949 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:34.775920 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252\": container with ID starting with ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252 not found: ID does not exist" containerID="ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252" Nov 02 23:19:34.776066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.775963 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252"} err="failed to get container status \"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252\": rpc error: code = NotFound desc = could not find container \"ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252\": container with ID starting with ec399bfc28702b36ef24b5982a5a0a2c2f38b0851e56979fe9aa112594e7a252 not found: ID does not exist" Nov 02 23:19:34.808512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.802534 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msrn\" (UniqueName: \"kubernetes.io/projected/31c204dc-7776-4086-9a41-e57fbb4af357-kube-api-access-9msrn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:34.832764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.832740 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:19:34.903453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.903425 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\" (UniqueName: \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\") pod \"db35a20a-2cd9-4349-a6d5-314422a061a4\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " Nov 02 23:19:34.903592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.903550 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\") pod \"db35a20a-2cd9-4349-a6d5-314422a061a4\" (UID: \"db35a20a-2cd9-4349-a6d5-314422a061a4\") " Nov 02 23:19:34.903890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.903857 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume" pod "db35a20a-2cd9-4349-a6d5-314422a061a4" (UID: "db35a20a-2cd9-4349-a6d5-314422a061a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:34.904122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.904103 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db35a20a-2cd9-4349-a6d5-314422a061a4-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:34.905724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.905705 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl" pod "db35a20a-2cd9-4349-a6d5-314422a061a4" (UID: "db35a20a-2cd9-4349-a6d5-314422a061a4"). InnerVolumeSpecName "kube-api-access-ftcsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:34.913010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.912756 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:19:34.916135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:34.916115 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-048ef3-bf4c7666f-9mq9n"] Nov 02 23:19:35.005316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.005289 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftcsl\" (UniqueName: \"kubernetes.io/projected/db35a20a-2cd9-4349-a6d5-314422a061a4-kube-api-access-ftcsl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:35.031465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.031369 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" podStartSLOduration=2.601671232 podStartE2EDuration="5.031352653s" totalImagesPullingTime="2.429681421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:35.030619933 +0000 UTC m=+1889.460489999" watchObservedRunningTime="2026-11-02 23:19:35.031352653 +0000 UTC m=+1889.461222515" Nov 02 23:19:35.158318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.156615 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" podStartSLOduration=2.04387715 podStartE2EDuration="4.156587332s" totalImagesPullingTime="2.112710182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:35.147853451 +0000 UTC m=+1889.577723313" watchObservedRunningTime="2026-11-02 23:19:35.156587332 +0000 UTC m=+1889.586457210" Nov 02 23:19:35.694557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.694516 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c204dc-7776-4086-9a41-e57fbb4af357" path="/var/lib/kubelet/pods/31c204dc-7776-4086-9a41-e57fbb4af357/volumes" Nov 02 23:19:35.771398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.771371 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" Nov 02 23:19:35.771563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.771399 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj" event={"ID":"db35a20a-2cd9-4349-a6d5-314422a061a4","Type":"ContainerDied","Data":"60f5c3278f0c0a3f672ecd815f93c32f05ebecde3b79a17cf9ecccea21e3d8f4"} Nov 02 23:19:35.771563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.771453 4494 scope.go:122] "RemoveContainer" containerID="fd9b79bb26bb2a9ac35afee4b5c92fa8c68f88b9ca30de45ec0a6fd3f217d18c" Nov 02 23:19:35.982679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:35.974229 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:19:36.000754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:36.000716 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-048ef3-65b594fdc5-ldptj"] Nov 02 23:19:37.059889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:37.059853 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:37.422069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:37.422027 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:37.695214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:37.695110 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db35a20a-2cd9-4349-a6d5-314422a061a4" path="/var/lib/kubelet/pods/db35a20a-2cd9-4349-a6d5-314422a061a4/volumes" Nov 02 23:19:39.017986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.017944 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:19:39.018569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018553 4494 cpu_manager.go:414] "containerMap: removing container" podUID="db35a20a-2cd9-4349-a6d5-314422a061a4" containerName="payload-pre-processing" Nov 02 23:19:39.018628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018569 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db35a20a-2cd9-4349-a6d5-314422a061a4" containerName="payload-pre-processing" Nov 02 23:19:39.018628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018583 4494 cpu_manager.go:414] "containerMap: removing container" podUID="31c204dc-7776-4086-9a41-e57fbb4af357" containerName="payload-processing" Nov 02 23:19:39.018628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018588 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c204dc-7776-4086-9a41-e57fbb4af357" containerName="payload-processing" Nov 02 23:19:39.018723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018714 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="db35a20a-2cd9-4349-a6d5-314422a061a4" containerName="payload-pre-processing" Nov 02 23:19:39.018756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.018723 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c204dc-7776-4086-9a41-e57fbb4af357" containerName="payload-processing" Nov 02 23:19:39.023525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.023504 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.116791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.116760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.116943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.116896 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.116986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.116964 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsn2\" (UniqueName: \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.117082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.117058 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.127531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.127509 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2b1c72-serving-cert\"" Nov 02 23:19:39.156689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.130803 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:19:39.217545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.217507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:39.217852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.217832 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsn2\" (UniqueName: \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.217919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.217870 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.217981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.217947 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.218051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.218034 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.218102 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:39.218047 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-serving-cert: secret "maas-api-e2e-ipp-b-2b1c72-serving-cert" not found Nov 02 23:19:39.218157 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:39.218106 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls podName:275aba89-b42e-44b6-abe6-a32c3db6ff13 nodeName:}" failed. No retries permitted until 2026-11-02 23:19:39.718088757 +0000 UTC m=+1894.147958616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls") pod "maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" (UID: "275aba89-b42e-44b6-abe6-a32c3db6ff13") : secret "maas-api-e2e-ipp-b-2b1c72-serving-cert" not found Nov 02 23:19:39.218712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.218692 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.220609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.220587 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.313439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.308202 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsn2\" (UniqueName: \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.455629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.455576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:39.727268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.727238 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.729774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.729748 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:39.756153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.756110 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:39.952436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:39.952400 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:40.363270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:40.359802 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:19:40.981119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:40.981081 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" event={"ID":"275aba89-b42e-44b6-abe6-a32c3db6ff13","Type":"ContainerStarted","Data":"0907ca0bb900e6a852f2456596af830046ffcb2c31d83c74dfc687db3f21f7cd"} Nov 02 23:19:41.755845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:41.755813 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:19:41.756646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:41.756625 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:19:41.992256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:41.992224 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" event={"ID":"275aba89-b42e-44b6-abe6-a32c3db6ff13","Type":"ContainerStarted","Data":"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a"} Nov 02 23:19:41.992446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:41.992371 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:42.124214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:42.124086 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" podStartSLOduration=3.293198374 podStartE2EDuration="4.124072839s" totalImagesPullingTime="830.874465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:42.121233774 +0000 UTC m=+1896.551103636" watchObservedRunningTime="2026-11-02 23:19:42.124072839 +0000 UTC m=+1896.553942701" Nov 02 23:19:43.036851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:43.036808 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:43.185799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:43.185759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:44.618041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:44.618008 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:44.825260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:44.825227 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:45.088232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:45.088164 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:19:45.088466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:45.088440 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" containerID="cri-o://69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd" gracePeriod=30 Nov 02 23:19:45.096007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:45.095969 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:45.774983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:45.774951 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:19:45.775527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:45.775035 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:19:46.037835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:46.037738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:47.027384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.027292 4494 generic.go:356] "Generic (PLEG): container finished" podID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerID="94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18" exitCode=0 Nov 02 23:19:47.027384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.027333 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerDied","Data":"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18"} Nov 02 23:19:47.027967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.027852 4494 scope.go:122] "RemoveContainer" containerID="94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18" Nov 02 23:19:47.029613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.029580 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:47.029730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.029653 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:47.029730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.029685 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:47.083122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.083098 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:19:47.088481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.088462 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.141818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.141780 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1cb1f8-openshift-default-dockercfg-97hkb\"" Nov 02 23:19:47.177067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.177032 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192055 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192118 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192201 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192256 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192334 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skhk\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192382 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192406 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192430 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192474 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.193110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.192585 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:19:47.294247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294148 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294231 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294283 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294369 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294419 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294490 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294547 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294631 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skhk\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294643 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294682 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.294982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294908 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.295042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.294989 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.295042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.295020 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.295537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.295493 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.297554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.297532 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.297733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.297714 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.394027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.393906 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.394215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.394121 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skhk\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\") pod \"e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.405122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.405091 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:47.576636 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:47.576568 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:19:47.643222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.639339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:47.861093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.861062 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:19:47.868568 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:47.864746 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3c57a2_3a63_4fd7_ab9c_32ab486ceec7.slice/crio-b9da4916f60bd193ce73ac6362217e406881250b7f9f0abba6f1c3339056e7fe WatchSource:0}: Error finding container b9da4916f60bd193ce73ac6362217e406881250b7f9f0abba6f1c3339056e7fe: Status 404 returned error can't find the container with id b9da4916f60bd193ce73ac6362217e406881250b7f9f0abba6f1c3339056e7fe Nov 02 23:19:47.868568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.867712 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:47.868568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.867773 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:47.868568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:47.867814 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:19:48.004983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.004954 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:19:48.069712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.069671 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerStarted","Data":"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f"} Nov 02 23:19:48.071859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.071610 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerStarted","Data":"9054605860a9d6503d94ea930fef2f89bd0e9327b44b8bd6d8fd8b85a2594a3c"} Nov 02 23:19:48.071859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.071653 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerStarted","Data":"b9da4916f60bd193ce73ac6362217e406881250b7f9f0abba6f1c3339056e7fe"} Nov 02 23:19:48.270927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.270856 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podStartSLOduration=1.270833176 podStartE2EDuration="1.270833176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:48.249803568 +0000 UTC m=+1902.679673430" watchObservedRunningTime="2026-11-02 23:19:48.270833176 +0000 UTC m=+1902.700703041" Nov 02 23:19:48.414995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.413540 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:48.417000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.416961 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:19:48.417157 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.417023 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:19:48.780192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.780142 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:19:48.781405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.781378 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:19:48.781553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:48.781424 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 23:19:49.011435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:49.011404 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:19:49.410935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:49.410904 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:49.781317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:49.781231 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:19:49.781317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:49.781287 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 23:19:50.093446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.093365 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:50.094365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.094342 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:19:50.356395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.346907 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:50.526232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.526137 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:19:50.581869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.581830 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.581893 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.581980 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582054 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582122 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582242 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582311 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582366 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582408 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\") pod \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\" (UID: \"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b\") " Nov 02 23:19:50.582617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582592 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:50.582799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582766 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:50.582877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.582862 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:50.583317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.583292 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.583419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.583319 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.583419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.583351 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.583544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.583507 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:50.583544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.583513 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:50.585620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.585586 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:50.585848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.585831 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "kube-api-access-45cnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:50.585933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.585894 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:50.586644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.586619 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo" pod "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" (UID: "6e395c83-78c6-4ea4-8d62-ba136f8f3e7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:50.684443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684399 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45cnc\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-kube-api-access-45cnc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.684443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684441 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.684650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684458 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.684650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684471 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.684650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684484 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.684650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.684497 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:19:50.780343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.780305 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:19:50.780518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:50.780373 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 23:19:51.056249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.055537 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:51.101408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.101377 4494 generic.go:356] "Generic (PLEG): container finished" podID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerID="69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd" exitCode=0 Nov 02 23:19:51.101577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.101436 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" Nov 02 23:19:51.101577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.101463 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerDied","Data":"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd"} Nov 02 23:19:51.101577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.101530 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7" event={"ID":"6e395c83-78c6-4ea4-8d62-ba136f8f3e7b","Type":"ContainerDied","Data":"ea1c4cf96a9b8c9fc3788fd47859e4488ac8946a44a6ba5ac29ade9f93a06183"} Nov 02 23:19:51.101577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.101555 4494 scope.go:122] "RemoveContainer" containerID="69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd" Nov 02 23:19:51.113242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.113222 4494 scope.go:122] "RemoveContainer" containerID="225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3" Nov 02 23:19:51.126775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.126751 4494 scope.go:122] "RemoveContainer" containerID="69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd" Nov 02 23:19:51.127111 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:51.127092 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd\": container with ID starting with 69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd not found: ID does not exist" containerID="69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd" Nov 02 23:19:51.127244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.127122 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd"} err="failed to get container status \"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd\": rpc error: code = NotFound desc = could not find container \"69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd\": container with ID starting with 69d4c7588fe664f7446b583bbd52241b04ccecc4d706bf8e1ced6cc269098cdd not found: ID does not exist" Nov 02 23:19:51.127244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.127140 4494 scope.go:122] "RemoveContainer" containerID="225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3" Nov 02 23:19:51.127543 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:19:51.127512 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3\": container with ID starting with 225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3 not found: ID does not exist" containerID="225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3" Nov 02 23:19:51.127597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.127552 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3"} err="failed to get container status \"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3\": rpc error: code = NotFound desc = could not find container \"225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3\": container with ID starting with 225bb995b514e9bbf16108102eac7055d04ff4ff245fb919bb5d346ccb3a18b3 not found: ID does not exist" Nov 02 23:19:51.287298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.287257 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:19:51.310380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.310290 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:51.357407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.357370 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-048ef3-openshift-default-5cf6cb887-ftvp7"] Nov 02 23:19:51.467677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.467643 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:51.694437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.694407 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" path="/var/lib/kubelet/pods/6e395c83-78c6-4ea4-8d62-ba136f8f3e7b/volumes" Nov 02 23:19:51.781003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.780964 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:19:51.781163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:51.781029 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 23:19:52.781017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:52.780979 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 23:19:52.781482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:52.781043 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 23:19:53.784648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:53.784614 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:19:53.785744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:53.785722 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:19:54.348619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.348585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:54.557863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.557791 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:54.768745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.768709 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:19:54.769383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.769369 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:54.769437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.769385 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:54.769437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.769400 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:54.769437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.769405 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:54.769532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.769510 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:54.775359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.775336 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:54.828555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.828522 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1cb1f8\"" Nov 02 23:19:54.829108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.828683 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1cb1f8-dockercfg-zj25w\"" Nov 02 23:19:54.829282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.829259 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:19:54.969200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.968025 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:54.969200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:54.968292 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gcz\" (UniqueName: \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.028151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.025250 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:19:55.028151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.026543 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e395c83-78c6-4ea4-8d62-ba136f8f3e7b" containerName="istio-proxy" Nov 02 23:19:55.030841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.030808 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:19:55.031012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.030938 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.069736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.069705 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.069891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.069795 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgrd\" (UniqueName: \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.069891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.069820 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gcz\" (UniqueName: \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.069992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.069921 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.070380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.070364 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.131425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.131395 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gcz\" (UniqueName: \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.170615 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.170583 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.170785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.170729 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgrd\" (UniqueName: \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.171332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.171307 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.232679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.232651 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgrd\" (UniqueName: \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\") pod \"payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.351768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.351669 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:55.390771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.390740 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:19:55.733073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.733044 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:19:55.733476 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:55.733455 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef04ff2b_0aa6_4d5d_aed6_4520c75574ed.slice/crio-c640de0cbe182c9afa2e24d2a0a73ef84a539506780c35ed5ee093ba5426f0d0 WatchSource:0}: Error finding container c640de0cbe182c9afa2e24d2a0a73ef84a539506780c35ed5ee093ba5426f0d0: Status 404 returned error can't find the container with id c640de0cbe182c9afa2e24d2a0a73ef84a539506780c35ed5ee093ba5426f0d0 Nov 02 23:19:55.834049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.834025 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:19:55.834720 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:19:55.834692 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada5245d_2663_4686_a035_e5fbd7bf4e8f.slice/crio-80e2ac6d01a08dc50c7ea286f6a07c356cddff9494032a143296723086dfc411 WatchSource:0}: Error finding container 80e2ac6d01a08dc50c7ea286f6a07c356cddff9494032a143296723086dfc411: Status 404 returned error can't find the container with id 80e2ac6d01a08dc50c7ea286f6a07c356cddff9494032a143296723086dfc411 Nov 02 23:19:55.913796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.913767 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:19:55.944034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.944001 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:19:55.944255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:55.944220 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" containerID="cri-o://6ccf2baebfae2408b577abde90e4be39a6e5fe9cfa0f5ab39ef7b1dbbafba5d0" gracePeriod=30 Nov 02 23:19:56.140847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.140741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" event={"ID":"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed","Type":"ContainerStarted","Data":"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7"} Nov 02 23:19:56.141031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.140857 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" event={"ID":"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed","Type":"ContainerStarted","Data":"c640de0cbe182c9afa2e24d2a0a73ef84a539506780c35ed5ee093ba5426f0d0"} Nov 02 23:19:56.141347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.141258 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:19:56.148810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.148720 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" event={"ID":"ada5245d-2663-4686-a035-e5fbd7bf4e8f","Type":"ContainerStarted","Data":"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f"} Nov 02 23:19:56.149096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.149071 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" event={"ID":"ada5245d-2663-4686-a035-e5fbd7bf4e8f","Type":"ContainerStarted","Data":"80e2ac6d01a08dc50c7ea286f6a07c356cddff9494032a143296723086dfc411"} Nov 02 23:19:56.260355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:56.260298 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" podStartSLOduration=2.260282893 podStartE2EDuration="2.260282893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:56.257577253 +0000 UTC m=+1910.687447114" watchObservedRunningTime="2026-11-02 23:19:56.260282893 +0000 UTC m=+1910.690152760" Nov 02 23:19:57.155734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:19:57.155700 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:20:00.391339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:00.391306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:01.193204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.193086 4494 generic.go:356] "Generic (PLEG): container finished" podID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerID="6ccf2baebfae2408b577abde90e4be39a6e5fe9cfa0f5ab39ef7b1dbbafba5d0" exitCode=0 Nov 02 23:20:01.193395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.193344 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" event={"ID":"f5987d37-eda3-4816-a39e-3bd4229da31a","Type":"ContainerDied","Data":"6ccf2baebfae2408b577abde90e4be39a6e5fe9cfa0f5ab39ef7b1dbbafba5d0"} Nov 02 23:20:01.299245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.298294 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" podStartSLOduration=7.298274379 podStartE2EDuration="7.298274379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:56.3996119 +0000 UTC m=+1910.829481761" watchObservedRunningTime="2026-11-02 23:20:01.298274379 +0000 UTC m=+1915.728144240" Nov 02 23:20:01.304198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.301650 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:20:01.373779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.373747 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:20:01.487588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487552 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487615 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487699 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487738 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487771 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487811 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487851 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.487944 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488006 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:01.488495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488039 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488063 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:01.488495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488087 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:01.488495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488135 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\") pod \"f5987d37-eda3-4816-a39e-3bd4229da31a\" (UID: \"f5987d37-eda3-4816-a39e-3bd4229da31a\") " Nov 02 23:20:01.488784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.488731 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:01.489062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.489039 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987d37-eda3-4816-a39e-3bd4229da31a-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.489186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.489070 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.489186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.489087 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.489186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.489105 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.489186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.489120 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.490164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.490140 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:01.490297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.490276 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "kube-api-access-gxtst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:01.490453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.490435 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:01.490533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.490508 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy" pod "f5987d37-eda3-4816-a39e-3bd4229da31a" (UID: "f5987d37-eda3-4816-a39e-3bd4229da31a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:01.590311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.590230 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.590311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.590266 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtst\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-kube-api-access-gxtst\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.590311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.590278 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:01.590311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:01.590293 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987d37-eda3-4816-a39e-3bd4229da31a-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:02.133135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.133094 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:02.200856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.200821 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" event={"ID":"f5987d37-eda3-4816-a39e-3bd4229da31a","Type":"ContainerDied","Data":"84c0809d0efe57eb372192ad1a9879a3a96bb35096d6c32529d8ff1d4805e28f"} Nov 02 23:20:02.200856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.200845 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5" Nov 02 23:20:02.201073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.200926 4494 scope.go:122] "RemoveContainer" containerID="6ccf2baebfae2408b577abde90e4be39a6e5fe9cfa0f5ab39ef7b1dbbafba5d0" Nov 02 23:20:02.292209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.289358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:02.384450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.384365 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:20:02.388843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.388817 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-dkzf5"] Nov 02 23:20:02.599396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.599368 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:20:02.599809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.599600 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" containerID="cri-o://8db44ae5e8e08bd484bc59437356093ef0f827a382f30d3a17df6376e0dfe50b" gracePeriod=30 Nov 02 23:20:02.632087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:02.632025 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:03.429511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.429483 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:03.694717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.694620 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" path="/var/lib/kubelet/pods/f5987d37-eda3-4816-a39e-3bd4229da31a/volumes" Nov 02 23:20:03.962016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.961939 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:20:03.963025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.963007 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" Nov 02 23:20:03.963090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.963029 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" Nov 02 23:20:03.963231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.963218 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5987d37-eda3-4816-a39e-3bd4229da31a" containerName="istio-proxy" Nov 02 23:20:03.968595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:03.968574 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.031304 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eb3c46-openshift-default-dockercfg-5qs5p\"" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032223 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032377 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032473 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhcn\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032512 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032565 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032615 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032746 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.033210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.032780 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.054539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.054501 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.155987 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhcn\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156052 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156110 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156162 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156263 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156319 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156351 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156393 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.216285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.156464 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.221340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.221128 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.222648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.221671 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.222648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.221901 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.222648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.222122 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.224026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.224007 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:20:04.227341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.227253 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.280242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.280149 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.282653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.282604 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.287416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.287388 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.291039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.291010 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhcn\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\") pod \"e2e-api-a-eb3c46-openshift-default-587b477687-lzg22\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:04.420141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.420104 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:04.585375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:04.585288 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:05.060530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.060449 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:20:05.061159 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:05.061117 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841753d8_819a_4821_b6d9_df462630262b.slice/crio-902d29b92942b898c6337bbe8615243666c64a39879238eda11fb08a6fb13f06 WatchSource:0}: Error finding container 902d29b92942b898c6337bbe8615243666c64a39879238eda11fb08a6fb13f06: Status 404 returned error can't find the container with id 902d29b92942b898c6337bbe8615243666c64a39879238eda11fb08a6fb13f06 Nov 02 23:20:05.063904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.063875 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:05.063983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.063939 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:05.063983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.063971 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:05.274858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.274689 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerStarted","Data":"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5"} Nov 02 23:20:05.274858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.274741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerStarted","Data":"902d29b92942b898c6337bbe8615243666c64a39879238eda11fb08a6fb13f06"} Nov 02 23:20:05.399328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.399197 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podStartSLOduration=2.399157472 podStartE2EDuration="2.399157472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:05.397519533 +0000 UTC m=+1919.827389394" watchObservedRunningTime="2026-11-02 23:20:05.399157472 +0000 UTC m=+1919.829027341" Nov 02 23:20:05.585427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:05.585340 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:06.156233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:06.156153 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:06.307018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:06.306979 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:06.591979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:06.591948 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:07.056645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.056612 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4"] Nov 02 23:20:07.059607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.059576 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3281789d-nzpz4"] Nov 02 23:20:07.159188 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.159139 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:20:07.290459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.290430 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:07.291703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.291681 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:20:07.695409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.695371 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a9a51a-6a30-468f-b677-876097bed8aa" path="/var/lib/kubelet/pods/e9a9a51a-6a30-468f-b677-876097bed8aa/volumes" Nov 02 23:20:07.799679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.799650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:07.993320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:07.993287 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:20:08.094686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.094648 4494 scope.go:122] "RemoveContainer" containerID="d30bb4641382cbfbc16be521b4dc36a157a2a612c22ddb4a4b78eb987a2d8455" Nov 02 23:20:08.104990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.104971 4494 scope.go:122] "RemoveContainer" containerID="8db44ae5e8e08bd484bc59437356093ef0f827a382f30d3a17df6376e0dfe50b" Nov 02 23:20:08.110274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110246 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110288 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110327 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110382 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110416 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110460 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110541 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110557 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:08.110662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110569 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:08.110662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110643 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110723 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\") pod \"b3ff9079-a428-4714-ad29-75ed3beb3768\" (UID: \"b3ff9079-a428-4714-ad29-75ed3beb3768\") " Nov 02 23:20:08.110889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110797 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:08.110889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.110805 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:08.111236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.111214 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:08.111440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.111415 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.111597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.111580 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.111935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.111917 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.112077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.112063 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.113064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.113029 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "kube-api-access-vvgh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:08.113152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.113131 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:08.113476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.113453 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:08.117297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.117273 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo" pod "b3ff9079-a428-4714-ad29-75ed3beb3768" (UID: "b3ff9079-a428-4714-ad29-75ed3beb3768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:20:08.165831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.165809 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:20:08.213897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.213815 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.213897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.213842 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff9079-a428-4714-ad29-75ed3beb3768-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.213897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.213853 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.213897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.213862 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ff9079-a428-4714-ad29-75ed3beb3768-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.213897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.213872 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgh9\" (UniqueName: \"kubernetes.io/projected/b3ff9079-a428-4714-ad29-75ed3beb3768-kube-api-access-vvgh9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:08.291413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.291348 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:08.297131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.297096 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerDied","Data":"8db44ae5e8e08bd484bc59437356093ef0f827a382f30d3a17df6376e0dfe50b"} Nov 02 23:20:08.297301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.297189 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" event={"ID":"b3ff9079-a428-4714-ad29-75ed3beb3768","Type":"ContainerDied","Data":"421f490306a18571ccb0d9b8b65ba61631bb229ae57d9bea4602b440fb16a68a"} Nov 02 23:20:08.297301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.297113 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6" Nov 02 23:20:08.457791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.457754 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:08.573265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.573158 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:20:08.577856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:08.577829 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a7a67875-openshift-default-7465746989-fp4c6"] Nov 02 23:20:09.696387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:09.696347 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" path="/var/lib/kubelet/pods/b3ff9079-a428-4714-ad29-75ed3beb3768/volumes" Nov 02 23:20:10.238445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:10.238394 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:20:10.238633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:10.238460 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:20:11.440036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:11.440006 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:11.619459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:11.619422 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:11.782867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:11.782738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:12.006834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.006789 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:20:12.008021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.007988 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.008242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.008222 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.030367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.030336 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.030567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.030549 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.030948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.030927 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.037812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.037740 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.087941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.087907 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:20:12.124426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.124389 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eb3c46-dockercfg-qb9kq\"" Nov 02 23:20:12.168427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.168401 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eb3c46\"" Nov 02 23:20:12.175257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.175227 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:20:12.175920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.175907 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ff9079-a428-4714-ad29-75ed3beb3768" containerName="istio-proxy" Nov 02 23:20:12.177078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.177052 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9k6\" (UniqueName: \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.177248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.177219 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.179741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.179722 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.241352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.241262 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:20:12.274891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.274857 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:12.278872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.278840 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.279019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.278934 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2k4l\" (UniqueName: \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.279120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.279097 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.279228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.279209 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9k6\" (UniqueName: \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.279605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.279588 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.370889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.370796 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9k6\" (UniqueName: \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\") pod \"payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.380318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.380285 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2k4l\" (UniqueName: \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.380472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.380429 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.381124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.381099 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.404717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.404396 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:12.442279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.442229 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2k4l\" (UniqueName: \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\") pod \"payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.556986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.504893 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:12.806732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.806703 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:20:12.808546 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:12.808521 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96a5f15_0ed9_42d3_8295_a27bb2980a78.slice/crio-58102166d0c560a8d42432f90e8d703f590903177cbb112d1694d335a44a04e2 WatchSource:0}: Error finding container 58102166d0c560a8d42432f90e8d703f590903177cbb112d1694d335a44a04e2: Status 404 returned error can't find the container with id 58102166d0c560a8d42432f90e8d703f590903177cbb112d1694d335a44a04e2 Nov 02 23:20:12.974706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:12.974676 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:20:12.975265 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:12.975240 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62009735_e9ab_40c2_aec8_6fe861465d58.slice/crio-380b5c15ad00dc4f7d98b74f292f82d2481b35de7b2c2e2d75b9ecbd06e00a57 WatchSource:0}: Error finding container 380b5c15ad00dc4f7d98b74f292f82d2481b35de7b2c2e2d75b9ecbd06e00a57: Status 404 returned error can't find the container with id 380b5c15ad00dc4f7d98b74f292f82d2481b35de7b2c2e2d75b9ecbd06e00a57 Nov 02 23:20:13.336765 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.336716 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" event={"ID":"f96a5f15-0ed9-42d3-8295-a27bb2980a78","Type":"ContainerStarted","Data":"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b"} Nov 02 23:20:13.336949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.336782 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" event={"ID":"f96a5f15-0ed9-42d3-8295-a27bb2980a78","Type":"ContainerStarted","Data":"58102166d0c560a8d42432f90e8d703f590903177cbb112d1694d335a44a04e2"} Nov 02 23:20:13.336949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.336852 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:13.338803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.338780 4494 generic.go:356] "Generic (PLEG): container finished" podID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerID="612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405" exitCode=0 Nov 02 23:20:13.338932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.338859 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerDied","Data":"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405"} Nov 02 23:20:13.339276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.339258 4494 scope.go:122] "RemoveContainer" containerID="612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405" Nov 02 23:20:13.340618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340598 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" event={"ID":"62009735-e9ab-40c2-aec8-6fe861465d58","Type":"ContainerStarted","Data":"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5"} Nov 02 23:20:13.340743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340644 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" event={"ID":"62009735-e9ab-40c2-aec8-6fe861465d58","Type":"ContainerStarted","Data":"380b5c15ad00dc4f7d98b74f292f82d2481b35de7b2c2e2d75b9ecbd06e00a57"} Nov 02 23:20:13.340743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340678 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:13.340838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340786 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:13.340877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340847 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:13.340915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.340879 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:13.345595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.345578 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:20:13.350775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.350759 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.404201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.404142 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4bf70\"" Nov 02 23:20:13.404340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.404314 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4bf70-dockercfg-vxthw\"" Nov 02 23:20:13.407347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.407324 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:20:13.427234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.426675 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfrs\" (UniqueName: \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.427234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.426995 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.454028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.453979 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" podStartSLOduration=2.45396662 podStartE2EDuration="2.45396662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:13.453086999 +0000 UTC m=+1927.882956860" watchObservedRunningTime="2026-11-02 23:20:13.45396662 +0000 UTC m=+1927.883836481" Nov 02 23:20:13.528733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.528699 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.528939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.528847 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfrs\" (UniqueName: \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.529617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.529592 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.639955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.639866 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfrs\" (UniqueName: \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\") pod \"payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.710013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.709984 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:13.853473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:13.853405 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" podStartSLOduration=1.8533849980000001 podStartE2EDuration="1.853384998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:13.851730216 +0000 UTC m=+1928.281600083" watchObservedRunningTime="2026-11-02 23:20:13.853384998 +0000 UTC m=+1928.283254860" Nov 02 23:20:14.070848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.070816 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:20:14.076367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.076349 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.106976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.106945 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:20:14.107292 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:14.107268 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5b29cf_27e4_4b89_b5c0_07a8055534ae.slice/crio-04acf58faf307e061be26a182e860150525725f2b2a9c016053996d263ccf466 WatchSource:0}: Error finding container 04acf58faf307e061be26a182e860150525725f2b2a9c016053996d263ccf466: Status 404 returned error can't find the container with id 04acf58faf307e061be26a182e860150525725f2b2a9c016053996d263ccf466 Nov 02 23:20:14.132682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.132633 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:20:14.146061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.146030 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khh9k\" (UniqueName: \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.146289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.146101 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.247920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.247887 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khh9k\" (UniqueName: \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.248084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.247945 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.248674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.248653 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.309277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.309246 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khh9k\" (UniqueName: \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\") pod \"payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.349738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.349652 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" event={"ID":"aa5b29cf-27e4-4b89-b5c0-07a8055534ae","Type":"ContainerStarted","Data":"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa"} Nov 02 23:20:14.349738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.349712 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" event={"ID":"aa5b29cf-27e4-4b89-b5c0-07a8055534ae","Type":"ContainerStarted","Data":"04acf58faf307e061be26a182e860150525725f2b2a9c016053996d263ccf466"} Nov 02 23:20:14.349955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.349741 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:14.351886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.351861 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerStarted","Data":"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13"} Nov 02 23:20:14.391309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.391286 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:14.476735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.476555 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" podStartSLOduration=1.476535403 podStartE2EDuration="1.476535403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:14.471573345 +0000 UTC m=+1928.901443206" watchObservedRunningTime="2026-11-02 23:20:14.476535403 +0000 UTC m=+1928.906405265" Nov 02 23:20:14.774648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.774618 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:20:14.775627 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:14.775601 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba90937f_b697_46f8_9709_d8a4d69c5c6b.slice/crio-085c342a3ffbd57e41ae3d86741bc4fda4b6504ca44150c29744ff4cace15127 WatchSource:0}: Error finding container 085c342a3ffbd57e41ae3d86741bc4fda4b6504ca44150c29744ff4cace15127: Status 404 returned error can't find the container with id 085c342a3ffbd57e41ae3d86741bc4fda4b6504ca44150c29744ff4cace15127 Nov 02 23:20:14.984509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.984474 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:20:14.984693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.984666 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:14.984742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:14.984719 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:15.144864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.144752 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:15.373472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.373435 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" event={"ID":"ba90937f-b697-46f8-9709-d8a4d69c5c6b","Type":"ContainerStarted","Data":"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5"} Nov 02 23:20:15.373672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.373494 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" event={"ID":"ba90937f-b697-46f8-9709-d8a4d69c5c6b","Type":"ContainerStarted","Data":"085c342a3ffbd57e41ae3d86741bc4fda4b6504ca44150c29744ff4cace15127"} Nov 02 23:20:15.373812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.373789 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:15.492509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.492445 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" podStartSLOduration=1.492428064 podStartE2EDuration="1.492428064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:15.49041432 +0000 UTC m=+1929.920284215" watchObservedRunningTime="2026-11-02 23:20:15.492428064 +0000 UTC m=+1929.922297931" Nov 02 23:20:15.984462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.984421 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:15.984643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:15.984499 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:16.984312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:16.984280 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:16.984781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:16.984331 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:17.589924 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:17.589826 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:20:17.984439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:17.984399 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:17.985002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:17.984465 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:18.984352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:18.984322 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:18.984555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:18.984375 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:19.984117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:19.984081 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 23:20:19.984329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:19.984131 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 23:20:20.196867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:20.196824 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:20.989290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:20.989258 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:20:20.989571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:20.989552 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:20:20.990225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:20.990202 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:20:23.076859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:23.076825 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:23.273288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:23.273209 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:23.828874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:23.828805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:24.040345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.040316 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:20:24.151782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.151698 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:20:24.355916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.355881 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:20:24.356079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.355983 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:20:24.407791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.407696 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:24.553642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.553609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:24.698557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:24.698525 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:25.028660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.028585 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:20:25.034631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.034610 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.049406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.049370 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.049721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.049692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nls\" (UniqueName: \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.050357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.050323 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.050528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.050484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.097209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.091000 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ee5b4-serving-cert\"" Nov 02 23:20:25.102220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.098827 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:20:25.152039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.152006 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nls\" (UniqueName: \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.152545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.152146 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.152545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.152294 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.152545 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:25.152339 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-serving-cert: secret "maas-api-e2e-ipp-b-0ee5b4-serving-cert" not found Nov 02 23:20:25.152545 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:25.152413 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls podName:a58cd826-b563-46db-9e66-857898fa9666 nodeName:}" failed. No retries permitted until 2026-11-02 23:20:25.652389968 +0000 UTC m=+1940.082259824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls") pod "maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" (UID: "a58cd826-b563-46db-9e66-857898fa9666") : secret "maas-api-e2e-ipp-b-0ee5b4-serving-cert" not found Nov 02 23:20:25.152545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.152438 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.153221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.153164 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.155320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.155298 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.211702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.211662 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:25.214496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.214467 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nls\" (UniqueName: \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.378250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.378037 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:25.386194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.382350 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:20:25.677509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.677478 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:20:25.688198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.683245 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708461 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbjx\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708548 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708627 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708656 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708704 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708751 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708799 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708845 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708874 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.709194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.708965 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.733041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.732760 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:25.790225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.782650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:25.790225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.789443 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:20:25.809796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.809756 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.809812 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.809936 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbjx\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.809997 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810087 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810114 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810191 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810232 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810257 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810324 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810572 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810677 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810783 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.810991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.810950 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.811586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.811568 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-92072f-openshift-default-dockercfg-4sdgv\"" Nov 02 23:20:25.839199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.814276 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.839199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.823507 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.910196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.906990 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.919205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.917142 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbjx\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\") pod \"e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:25.939329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:25.939221 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:26.007341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.006856 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:26.085932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.085439 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:26.091026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.090999 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:26.382745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.382674 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:20:26.390062 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:26.390036 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58cd826_b563_46db_9e66_857898fa9666.slice/crio-8b22af4063eb6aeb1773424c47e2de664fbb3ee4d7aa9d7e3cff85bff9ce6540 WatchSource:0}: Error finding container 8b22af4063eb6aeb1773424c47e2de664fbb3ee4d7aa9d7e3cff85bff9ce6540: Status 404 returned error can't find the container with id 8b22af4063eb6aeb1773424c47e2de664fbb3ee4d7aa9d7e3cff85bff9ce6540 Nov 02 23:20:26.390617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.390582 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:20:26.455894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.455861 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" event={"ID":"a58cd826-b563-46db-9e66-857898fa9666","Type":"ContainerStarted","Data":"8b22af4063eb6aeb1773424c47e2de664fbb3ee4d7aa9d7e3cff85bff9ce6540"} Nov 02 23:20:26.505582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.505539 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:20:26.505707 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:26.505656 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f8fad4_e3ae_492c_9719_cd76c0825085.slice/crio-670ad36bbee5b53498e240d47763c1b5cef84ff3aa1cc9e82a05563e48950b2b WatchSource:0}: Error finding container 670ad36bbee5b53498e240d47763c1b5cef84ff3aa1cc9e82a05563e48950b2b: Status 404 returned error can't find the container with id 670ad36bbee5b53498e240d47763c1b5cef84ff3aa1cc9e82a05563e48950b2b Nov 02 23:20:26.508854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.508820 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:26.508978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.508884 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:26.508978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:26.508910 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:20:27.463933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.463895 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" event={"ID":"a58cd826-b563-46db-9e66-857898fa9666","Type":"ContainerStarted","Data":"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe"} Nov 02 23:20:27.464418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.463967 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:27.465741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.465708 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerStarted","Data":"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005"} Nov 02 23:20:27.465872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.465762 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerStarted","Data":"670ad36bbee5b53498e240d47763c1b5cef84ff3aa1cc9e82a05563e48950b2b"} Nov 02 23:20:27.583888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.583790 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" podStartSLOduration=2.750415129 podStartE2EDuration="3.583776061s" totalImagesPullingTime="833.360932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:27.58270402 +0000 UTC m=+1942.012573882" watchObservedRunningTime="2026-11-02 23:20:27.583776061 +0000 UTC m=+1942.013645922" Nov 02 23:20:27.739095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:27.739029 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podStartSLOduration=2.739009485 podStartE2EDuration="2.739009485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:27.728564514 +0000 UTC m=+1942.158434382" watchObservedRunningTime="2026-11-02 23:20:27.739009485 +0000 UTC m=+1942.168879347" Nov 02 23:20:28.091567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:28.091539 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:28.097081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:28.097053 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:28.472574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:28.472542 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:28.473534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:28.473511 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:20:29.040707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.040669 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" podUID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" containerName="payload-processing" containerID="cri-o://b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8" gracePeriod=25 Nov 02 23:20:29.152162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.152123 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" podUID="dc88b847-04ff-446e-a440-a8584a69affb" containerName="payload-pre-processing" containerID="cri-o://6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912" gracePeriod=25 Nov 02 23:20:29.317432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.317406 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:20:29.380331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.380292 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\") pod \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " Nov 02 23:20:29.380525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.380383 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\" (UniqueName: \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\") pod \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\" (UID: \"d9ad77fd-737c-4fca-a18c-78a8469a8b05\") " Nov 02 23:20:29.381522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.381486 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume" pod "d9ad77fd-737c-4fca-a18c-78a8469a8b05" (UID: "d9ad77fd-737c-4fca-a18c-78a8469a8b05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:29.384218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.384182 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw" pod "d9ad77fd-737c-4fca-a18c-78a8469a8b05" (UID: "d9ad77fd-737c-4fca-a18c-78a8469a8b05"). InnerVolumeSpecName "kube-api-access-829xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:29.424647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.424620 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:20:29.481358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481323 4494 generic.go:356] "Generic (PLEG): container finished" podID="dc88b847-04ff-446e-a440-a8584a69affb" containerID="6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912" exitCode=0 Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481384 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481410 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" event={"ID":"dc88b847-04ff-446e-a440-a8584a69affb","Type":"ContainerDied","Data":"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912"} Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481471 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz" event={"ID":"dc88b847-04ff-446e-a440-a8584a69affb","Type":"ContainerDied","Data":"384441406d8128347a7f2442e3e7ac64316238a688fa15c3479c72c9d72d0b1b"} Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481520 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\") pod \"dc88b847-04ff-446e-a440-a8584a69affb\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481499 4494 scope.go:122] "RemoveContainer" containerID="6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912" Nov 02 23:20:29.481862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481749 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\" (UniqueName: \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\") pod \"dc88b847-04ff-446e-a440-a8584a69affb\" (UID: \"dc88b847-04ff-446e-a440-a8584a69affb\") " Nov 02 23:20:29.482212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.481926 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume" pod "dc88b847-04ff-446e-a440-a8584a69affb" (UID: "dc88b847-04ff-446e-a440-a8584a69affb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:29.482276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.482246 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ad77fd-737c-4fca-a18c-78a8469a8b05-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:29.482276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.482267 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc88b847-04ff-446e-a440-a8584a69affb-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:29.482382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.482278 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829xw\" (UniqueName: \"kubernetes.io/projected/d9ad77fd-737c-4fca-a18c-78a8469a8b05-kube-api-access-829xw\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:29.483214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.483148 4494 generic.go:356] "Generic (PLEG): container finished" podID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" containerID="b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8" exitCode=0 Nov 02 23:20:29.483326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.483221 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" event={"ID":"d9ad77fd-737c-4fca-a18c-78a8469a8b05","Type":"ContainerDied","Data":"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8"} Nov 02 23:20:29.483326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.483268 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" event={"ID":"d9ad77fd-737c-4fca-a18c-78a8469a8b05","Type":"ContainerDied","Data":"bb8eeb23078b5fab9b379ad732cdfbd1680a66f33c86b887054aac2ac9989af8"} Nov 02 23:20:29.483326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.483270 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9" Nov 02 23:20:29.484591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.484567 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r" pod "dc88b847-04ff-446e-a440-a8584a69affb" (UID: "dc88b847-04ff-446e-a440-a8584a69affb"). InnerVolumeSpecName "kube-api-access-4s44r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:29.492594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.492579 4494 scope.go:122] "RemoveContainer" containerID="6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912" Nov 02 23:20:29.492872 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:29.492854 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912\": container with ID starting with 6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912 not found: ID does not exist" containerID="6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912" Nov 02 23:20:29.492924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.492882 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912"} err="failed to get container status \"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912\": rpc error: code = NotFound desc = could not find container \"6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912\": container with ID starting with 6b86dd753e5c37aaf638401e456489a1c7a2b2a6bbcb1795ffbf56d64c7e9912 not found: ID does not exist" Nov 02 23:20:29.492924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.492898 4494 scope.go:122] "RemoveContainer" containerID="b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8" Nov 02 23:20:29.506752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.506732 4494 scope.go:122] "RemoveContainer" containerID="b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8" Nov 02 23:20:29.507019 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:29.507001 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8\": container with ID starting with b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8 not found: ID does not exist" containerID="b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8" Nov 02 23:20:29.507097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.507027 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8"} err="failed to get container status \"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8\": rpc error: code = NotFound desc = could not find container \"b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8\": container with ID starting with b8df94d85f2f846f7fa6d6eee63465ea56c820166b9f364885e94138742ac4d8 not found: ID does not exist" Nov 02 23:20:29.583987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.583881 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s44r\" (UniqueName: \"kubernetes.io/projected/dc88b847-04ff-446e-a440-a8584a69affb-kube-api-access-4s44r\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:20:29.718007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.717965 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:20:29.723424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.723396 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-58ddcb7465-wq4v9"] Nov 02 23:20:29.898783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.898699 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:20:29.902040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:29.902012 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-74cc8c4fd7-g4jbz"] Nov 02 23:20:30.642521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:30.642484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:30.787763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:30.787720 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:30.935216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:30.935180 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:31.695409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:31.695374 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" path="/var/lib/kubelet/pods/d9ad77fd-737c-4fca-a18c-78a8469a8b05/volumes" Nov 02 23:20:31.695882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:31.695863 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc88b847-04ff-446e-a440-a8584a69affb" path="/var/lib/kubelet/pods/dc88b847-04ff-446e-a440-a8584a69affb/volumes" Nov 02 23:20:33.478625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:33.478596 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:20:35.703292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:35.703260 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:35.915632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:35.906322 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:36.020255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:36.017719 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:20:37.678606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:37.678572 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:42.846025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:42.845984 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:47.570456 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:47.570412 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:20:48.206460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:48.206424 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:53.277122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277090 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:20:53.277755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277740 4494 cpu_manager.go:414] "containerMap: removing container" podUID="dc88b847-04ff-446e-a440-a8584a69affb" containerName="payload-pre-processing" Nov 02 23:20:53.277807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277756 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc88b847-04ff-446e-a440-a8584a69affb" containerName="payload-pre-processing" Nov 02 23:20:53.277807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277790 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" containerName="payload-processing" Nov 02 23:20:53.277807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277795 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" containerName="payload-processing" Nov 02 23:20:53.277925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277884 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc88b847-04ff-446e-a440-a8584a69affb" containerName="payload-pre-processing" Nov 02 23:20:53.277925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.277906 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ad77fd-737c-4fca-a18c-78a8469a8b05" containerName="payload-processing" Nov 02 23:20:53.282969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.282951 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.312060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.312027 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.312262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.312241 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.312368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.312350 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.312431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.312389 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phr2t\" (UniqueName: \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.386440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.386410 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1cb1f8-serving-cert\"" Nov 02 23:20:53.390334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.390311 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:20:53.413624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.413589 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.413780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.413691 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.413780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.413720 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phr2t\" (UniqueName: \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.414037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.414019 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.414137 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:53.414124 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-serving-cert: secret "maas-api-e2e-ipp-a-1cb1f8-serving-cert" not found Nov 02 23:20:53.414230 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:20:53.414187 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls podName:518472cf-4dbe-45d1-a382-e0194c3af639 nodeName:}" failed. No retries permitted until 2026-11-02 23:20:53.914155331 +0000 UTC m=+1968.344025170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls") pod "maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" (UID: "518472cf-4dbe-45d1-a382-e0194c3af639") : secret "maas-api-e2e-ipp-a-1cb1f8-serving-cert" not found Nov 02 23:20:53.414527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.414505 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.416932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.416902 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.470952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.470922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:53.475938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.475910 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phr2t\" (UniqueName: \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.633606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.633527 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:53.783311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.783271 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:53.924760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.924728 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.927577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.927546 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:53.957385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:53.957350 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:54.197810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:54.197723 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:54.576433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:54.576400 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:20:54.576896 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:20:54.576868 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518472cf_4dbe_45d1_a382_e0194c3af639.slice/crio-fc1e51654600ac87c9b5741daafe3ad02e38b67f6f2c705560239615010ef8da WatchSource:0}: Error finding container fc1e51654600ac87c9b5741daafe3ad02e38b67f6f2c705560239615010ef8da: Status 404 returned error can't find the container with id fc1e51654600ac87c9b5741daafe3ad02e38b67f6f2c705560239615010ef8da Nov 02 23:20:54.664781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:54.664742 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" event={"ID":"518472cf-4dbe-45d1-a382-e0194c3af639","Type":"ContainerStarted","Data":"fc1e51654600ac87c9b5741daafe3ad02e38b67f6f2c705560239615010ef8da"} Nov 02 23:20:55.365769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.365745 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:20:55.477097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.477060 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:20:55.679214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.677312 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" event={"ID":"518472cf-4dbe-45d1-a382-e0194c3af639","Type":"ContainerStarted","Data":"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd"} Nov 02 23:20:55.679214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.677667 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:20:55.726655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.726621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:55.827519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.827450 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" podStartSLOduration=2.054289949 podStartE2EDuration="2.827428271s" totalImagesPullingTime="773.138322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:55.802977895 +0000 UTC m=+1970.232847762" watchObservedRunningTime="2026-11-02 23:20:55.827428271 +0000 UTC m=+1970.257298139" Nov 02 23:20:55.875758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:55.875727 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:56.041810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:56.041721 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:20:59.084297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:20:59.084239 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:00.366648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.366598 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" podUID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" containerName="payload-processing" containerID="cri-o://c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a" gracePeriod=25 Nov 02 23:21:00.478149 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.478072 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" podUID="5608db5a-403a-4263-89aa-c9e7c893bfd7" containerName="payload-pre-processing" containerID="cri-o://56fc654610cc5d77f64a41b88cb9105380f88af9f324f384c40cc6e92f33b536" gracePeriod=25 Nov 02 23:21:00.642020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.641987 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:21:00.724572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.724542 4494 generic.go:356] "Generic (PLEG): container finished" podID="5608db5a-403a-4263-89aa-c9e7c893bfd7" containerID="56fc654610cc5d77f64a41b88cb9105380f88af9f324f384c40cc6e92f33b536" exitCode=0 Nov 02 23:21:00.724769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.724647 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" event={"ID":"5608db5a-403a-4263-89aa-c9e7c893bfd7","Type":"ContainerDied","Data":"56fc654610cc5d77f64a41b88cb9105380f88af9f324f384c40cc6e92f33b536"} Nov 02 23:21:00.726384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.726360 4494 generic.go:356] "Generic (PLEG): container finished" podID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" containerID="c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a" exitCode=0 Nov 02 23:21:00.726537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.726431 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" Nov 02 23:21:00.726537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.726447 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" event={"ID":"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820","Type":"ContainerDied","Data":"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a"} Nov 02 23:21:00.726537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.726509 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt" event={"ID":"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820","Type":"ContainerDied","Data":"972d353c33841158413dd3bf5d282deb225a6c33cd911e606b18aec87e7a32ef"} Nov 02 23:21:00.726717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.726544 4494 scope.go:122] "RemoveContainer" containerID="c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a" Nov 02 23:21:00.736657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.736206 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\") pod \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " Nov 02 23:21:00.736657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.736265 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\" (UniqueName: \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\") pod \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\" (UID: \"43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820\") " Nov 02 23:21:00.736657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.736596 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume" pod "43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" (UID: "43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:00.736886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.736866 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:00.737805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.737785 4494 scope.go:122] "RemoveContainer" containerID="c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a" Nov 02 23:21:00.738128 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:00.738105 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a\": container with ID starting with c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a not found: ID does not exist" containerID="c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a" Nov 02 23:21:00.738199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.738136 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a"} err="failed to get container status \"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a\": rpc error: code = NotFound desc = could not find container \"c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a\": container with ID starting with c97144539154c665ed1fd3b489926cee6de17f77b57422c025c364b4ec7cac5a not found: ID does not exist" Nov 02 23:21:00.738621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.738600 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm" pod "43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" (UID: "43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820"). InnerVolumeSpecName "kube-api-access-8cbjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:00.761228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.761204 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:21:00.838514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.838481 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbjm\" (UniqueName: \"kubernetes.io/projected/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820-kube-api-access-8cbjm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:00.939978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.939945 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\" (UniqueName: \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\") pod \"5608db5a-403a-4263-89aa-c9e7c893bfd7\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " Nov 02 23:21:00.940132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.940058 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\") pod \"5608db5a-403a-4263-89aa-c9e7c893bfd7\" (UID: \"5608db5a-403a-4263-89aa-c9e7c893bfd7\") " Nov 02 23:21:00.940525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.940497 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume" pod "5608db5a-403a-4263-89aa-c9e7c893bfd7" (UID: "5608db5a-403a-4263-89aa-c9e7c893bfd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:00.940691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.940671 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5608db5a-403a-4263-89aa-c9e7c893bfd7-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:00.942548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:00.942519 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw" pod "5608db5a-403a-4263-89aa-c9e7c893bfd7" (UID: "5608db5a-403a-4263-89aa-c9e7c893bfd7"). InnerVolumeSpecName "kube-api-access-t5bkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:01.042476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.042447 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bkw\" (UniqueName: \"kubernetes.io/projected/5608db5a-403a-4263-89aa-c9e7c893bfd7-kube-api-access-t5bkw\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:01.149859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.149826 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:21:01.157714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.157687 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-695f5cc6cc-6mzlt"] Nov 02 23:21:01.694625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.694590 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" path="/var/lib/kubelet/pods/43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820/volumes" Nov 02 23:21:01.695205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.695161 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:21:01.735236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.735200 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" event={"ID":"5608db5a-403a-4263-89aa-c9e7c893bfd7","Type":"ContainerDied","Data":"0e957653dc8a6d6afcbcba63955487313cd772e9a63d5bc12a4bb18a712d32be"} Nov 02 23:21:01.735236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.735235 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9" Nov 02 23:21:01.735464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.735269 4494 scope.go:122] "RemoveContainer" containerID="56fc654610cc5d77f64a41b88cb9105380f88af9f324f384c40cc6e92f33b536" Nov 02 23:21:01.938372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.938151 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:21:01.942086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:01.942060 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-845b8fdcf5-m6ss9"] Nov 02 23:21:02.111766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:02.111139 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:02.435075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:02.435040 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:21:03.694524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:03.694492 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5608db5a-403a-4263-89aa-c9e7c893bfd7" path="/var/lib/kubelet/pods/5608db5a-403a-4263-89aa-c9e7c893bfd7/volumes" Nov 02 23:21:04.486497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:04.486461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:06.573252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:06.573216 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:08.206543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:08.206512 4494 scope.go:122] "RemoveContainer" containerID="2f2e8f2cbc143cd732836c8a01241ab0d0d2396283c9164feb4d193268f934f1" Nov 02 23:21:08.216675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:08.216653 4494 scope.go:122] "RemoveContainer" containerID="e7f2d3d6c9b81b5bce397bbfcd4b0659133cbfa997b9330b197bdf47ea64f5f7" Nov 02 23:21:09.861754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:09.861675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:13.822190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.822147 4494 generic.go:356] "Generic (PLEG): container finished" podID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerID="84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123" exitCode=0 Nov 02 23:21:13.822675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.822222 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerDied","Data":"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123"} Nov 02 23:21:13.822675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.822650 4494 scope.go:122] "RemoveContainer" containerID="84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123" Nov 02 23:21:13.824219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.824194 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:13.824289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.824264 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:13.824329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:13.824295 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:14.434231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.434195 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:21:14.434986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.434965 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5608db5a-403a-4263-89aa-c9e7c893bfd7" containerName="payload-pre-processing" Nov 02 23:21:14.434986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.434987 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5608db5a-403a-4263-89aa-c9e7c893bfd7" containerName="payload-pre-processing" Nov 02 23:21:14.435166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.435008 4494 cpu_manager.go:414] "containerMap: removing container" podUID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" containerName="payload-processing" Nov 02 23:21:14.435166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.435016 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" containerName="payload-processing" Nov 02 23:21:14.435166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.435163 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5608db5a-403a-4263-89aa-c9e7c893bfd7" containerName="payload-pre-processing" Nov 02 23:21:14.435166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.435201 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f6f7b4-7f8f-4f40-9d0e-f203ac4e3820" containerName="payload-processing" Nov 02 23:21:14.439004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.438982 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.492957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.492920 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-eb3c46-serving-cert\"" Nov 02 23:21:14.496654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.496628 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:21:14.544342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.544309 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztw7\" (UniqueName: \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.544536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.544511 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.544601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.544587 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.544667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.544653 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.634661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.634625 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:14.646193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.646145 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.646300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.646238 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.646300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.646281 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.646392 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:14.646377 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-serving-cert: secret "maas-api-e2e-api-a-eb3c46-serving-cert" not found Nov 02 23:21:14.646432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.646411 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztw7\" (UniqueName: \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.646467 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:14.646442 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls podName:e16251e0-34f7-4572-a885-a7ba6e525ab5 nodeName:}" failed. No retries permitted until 2026-11-02 23:21:15.146423049 +0000 UTC m=+1989.576292907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls") pod "maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" (UID: "e16251e0-34f7-4572-a885-a7ba6e525ab5") : secret "maas-api-e2e-api-a-eb3c46-serving-cert" not found Nov 02 23:21:14.647035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.646981 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.648782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.648764 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.725728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.725650 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztw7\" (UniqueName: \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:14.793371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.793333 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:14.843200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.838678 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerStarted","Data":"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818"} Nov 02 23:21:14.944724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.944650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:14.962285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.962243 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:21:14.962480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.962414 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:14.962480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:14.962466 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:15.051700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.051556 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m"] Nov 02 23:21:15.058551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.058514 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-847ba4df-4d48m"] Nov 02 23:21:15.156281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.156237 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:15.158840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.158811 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") pod \"maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:15.353149 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.352815 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:15.694796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.694765 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb8be88-5fbf-45dc-8925-1fb51e6829e0" path="/var/lib/kubelet/pods/feb8be88-5fbf-45dc-8925-1fb51e6829e0/volumes" Nov 02 23:21:15.738193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.734964 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:21:15.759113 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:21:15.759074 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16251e0_34f7_4572_a885_a7ba6e525ab5.slice/crio-f1d3a8d3110b9e5903027f92fbcaddda321512dac8eb25a6236320a012455832 WatchSource:0}: Error finding container f1d3a8d3110b9e5903027f92fbcaddda321512dac8eb25a6236320a012455832: Status 404 returned error can't find the container with id f1d3a8d3110b9e5903027f92fbcaddda321512dac8eb25a6236320a012455832 Nov 02 23:21:15.859956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.859916 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" event={"ID":"e16251e0-34f7-4572-a885-a7ba6e525ab5","Type":"ContainerStarted","Data":"f1d3a8d3110b9e5903027f92fbcaddda321512dac8eb25a6236320a012455832"} Nov 02 23:21:15.963430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.963353 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:15.963577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:15.963420 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:16.867809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:16.867776 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" event={"ID":"e16251e0-34f7-4572-a885-a7ba6e525ab5","Type":"ContainerStarted","Data":"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038"} Nov 02 23:21:16.868234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:16.867847 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:16.962708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:16.962673 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:16.962890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:16.962723 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:17.022362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:17.022260 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" podStartSLOduration=2.066329209 podStartE2EDuration="3.022246264s" totalImagesPullingTime="955.917055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:17.02195244 +0000 UTC m=+1991.451822302" watchObservedRunningTime="2026-11-02 23:21:17.022246264 +0000 UTC m=+1991.452116125" Nov 02 23:21:17.664276 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:17.664235 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:21:17.962777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:17.962676 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:17.962777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:17.962750 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:18.970923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:18.970886 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:18.971726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:18.971695 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:19.962091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:19.962060 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 23:21:19.962303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:19.962110 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 23:21:20.158656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:20.158592 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:20.968023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:20.967996 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:21:20.968437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:20.968420 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:21:20.969074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:20.969053 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:21:22.881861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:22.881832 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:21:26.931622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:26.931578 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:29.676856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:29.676819 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:29.823481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:29.823447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:30.038026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.037936 4494 generic.go:356] "Generic (PLEG): container finished" podID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerID="9054605860a9d6503d94ea930fef2f89bd0e9327b44b8bd6d8fd8b85a2594a3c" exitCode=0 Nov 02 23:21:30.038026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.038012 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerDied","Data":"9054605860a9d6503d94ea930fef2f89bd0e9327b44b8bd6d8fd8b85a2594a3c"} Nov 02 23:21:30.038474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.038460 4494 scope.go:122] "RemoveContainer" containerID="9054605860a9d6503d94ea930fef2f89bd0e9327b44b8bd6d8fd8b85a2594a3c" Nov 02 23:21:30.039980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.039954 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:30.040057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.040023 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:30.040092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.040055 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:30.461757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.461730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:30.948325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:30.948293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:31.054131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:31.054094 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerStarted","Data":"a09d99079e8304aeaa7aff46b9d7210c391e7c0138a224d993654138c825aa12"} Nov 02 23:21:31.418568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:31.413580 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:21:31.437536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:31.436873 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:31.437536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:31.436946 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:32.057832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.057792 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:21:32.062735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.062708 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.113890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.113862 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:21:32.160665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.160620 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d00785\"" Nov 02 23:21:32.160880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.160720 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d00785-dockercfg-ddzb4\"" Nov 02 23:21:32.162544 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.162507 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvm8d\" (UniqueName: \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.162730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.162706 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.264142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.264094 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm8d\" (UniqueName: \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.264386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.264236 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.264946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.264920 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.326160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.326087 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm8d\" (UniqueName: \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\") pod \"payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.377112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.377075 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:32.405913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.405871 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:32.406056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.405947 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:32.782457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.782429 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:21:32.783077 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:21:32.783051 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3391d6d7_a137_41cf_9144_181a86cb647c.slice/crio-d0505deef3885f12e9e82d2160d8b1391e55805bf99cce74225851f11319f7b1 WatchSource:0}: Error finding container d0505deef3885f12e9e82d2160d8b1391e55805bf99cce74225851f11319f7b1: Status 404 returned error can't find the container with id d0505deef3885f12e9e82d2160d8b1391e55805bf99cce74225851f11319f7b1 Nov 02 23:21:32.878230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.872235 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:21:32.884973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.884938 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:32.905597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.905567 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:21:32.976648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.976613 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2drd\" (UniqueName: \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:32.976808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:32.976752 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.073967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.073866 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" event={"ID":"3391d6d7-a137-41cf-9144-181a86cb647c","Type":"ContainerStarted","Data":"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1"} Nov 02 23:21:33.073967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.073925 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" event={"ID":"3391d6d7-a137-41cf-9144-181a86cb647c","Type":"ContainerStarted","Data":"d0505deef3885f12e9e82d2160d8b1391e55805bf99cce74225851f11319f7b1"} Nov 02 23:21:33.074433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.074026 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:33.077705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.077681 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2drd\" (UniqueName: \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.077839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.077787 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.078740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.078716 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.158251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.157982 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2drd\" (UniqueName: \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\") pod \"payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.192394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.192347 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" podStartSLOduration=1.192332523 podStartE2EDuration="1.192332523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:33.190538373 +0000 UTC m=+2007.620408260" watchObservedRunningTime="2026-11-02 23:21:33.192332523 +0000 UTC m=+2007.622202421" Nov 02 23:21:33.210139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.210104 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:33.411164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.409583 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:33.411164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.409644 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:33.619939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.619907 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:21:33.620827 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:21:33.620802 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e0952d_c068_44e2_ab73_c5ea24cad284.slice/crio-e7adde87892560ede90f6b19cd24a3d9e99edbd8098ef1abac0dd67e3fe2caba WatchSource:0}: Error finding container e7adde87892560ede90f6b19cd24a3d9e99edbd8098ef1abac0dd67e3fe2caba: Status 404 returned error can't find the container with id e7adde87892560ede90f6b19cd24a3d9e99edbd8098ef1abac0dd67e3fe2caba Nov 02 23:21:33.784674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:33.784631 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:34.082519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.082428 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" event={"ID":"a0e0952d-c068-44e2-ab73-c5ea24cad284","Type":"ContainerStarted","Data":"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b"} Nov 02 23:21:34.082519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.082491 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" event={"ID":"a0e0952d-c068-44e2-ab73-c5ea24cad284","Type":"ContainerStarted","Data":"e7adde87892560ede90f6b19cd24a3d9e99edbd8098ef1abac0dd67e3fe2caba"} Nov 02 23:21:34.083062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.082522 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:34.204903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.204852 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" podStartSLOduration=2.204839332 podStartE2EDuration="2.204839332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:34.202213979 +0000 UTC m=+2008.632083847" watchObservedRunningTime="2026-11-02 23:21:34.204839332 +0000 UTC m=+2008.634709211" Nov 02 23:21:34.406408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.406311 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:34.406408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:34.406363 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:35.093738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:35.093709 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:21:35.406240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:35.406127 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:35.406240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:35.406212 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:36.063206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:36.063140 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:21:36.405841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:36.405750 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 02 23:21:36.405841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:36.405822 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 02 23:21:37.410723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:37.410690 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:21:37.411831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:37.411808 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:21:38.810941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:38.810908 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:39.455220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:39.452114 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:39.595934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:39.595900 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:43.679507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:43.679458 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:43.910064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:43.908814 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:44.058118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:44.058010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:44.085146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:44.085113 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:21:45.091913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:45.091873 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:21:46.199771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.199694 4494 generic.go:356] "Generic (PLEG): container finished" podID="841753d8-819a-4821-b6d9-df462630262b" containerID="7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5" exitCode=0 Nov 02 23:21:46.200206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.199768 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerDied","Data":"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5"} Nov 02 23:21:46.200271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.200241 4494 scope.go:122] "RemoveContainer" containerID="7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5" Nov 02 23:21:46.201877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.201845 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:46.201957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.201926 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:46.201995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.201962 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:21:46.979328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:46.978035 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:47.218354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.218316 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerStarted","Data":"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19"} Nov 02 23:21:47.586436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.586260 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:21:47.587340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.587313 4494 patch_prober.go:29] interesting pod/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:21:47.587478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.587365 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 23:21:47.593650 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:47.593623 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:21:47.787079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.787038 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:21:47.881266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:47.879805 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:21:48.156949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.156846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:48.337201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.313089 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:48.565536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.565504 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:48.586094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.586058 4494 patch_prober.go:29] interesting pod/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:21:48.586273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.586122 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 23:21:48.759703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.759666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:48.964762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.964725 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:21:48.964988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.964956 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" podUID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" containerName="payload-processing" containerID="cri-o://c8adb2d62d9e92f9d77a691687360c12f45fa6330c65ac6e66978d7fae6ba907" gracePeriod=30 Nov 02 23:21:48.979610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.979579 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:21:48.979896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:48.979852 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" podUID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" containerName="payload-pre-processing" containerID="cri-o://5f42f40ccabd03dc8f730e234816139364687046ddef07bd43230025c15e62c2" gracePeriod=30 Nov 02 23:21:49.084232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.084097 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:49.227510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.227475 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:49.281881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.281853 4494 generic.go:356] "Generic (PLEG): container finished" podID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" containerID="c8adb2d62d9e92f9d77a691687360c12f45fa6330c65ac6e66978d7fae6ba907" exitCode=0 Nov 02 23:21:49.282021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.281929 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" event={"ID":"1d0e97d3-63b8-46b5-84cf-9be382585fcc","Type":"ContainerDied","Data":"c8adb2d62d9e92f9d77a691687360c12f45fa6330c65ac6e66978d7fae6ba907"} Nov 02 23:21:49.282021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.281987 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" event={"ID":"1d0e97d3-63b8-46b5-84cf-9be382585fcc","Type":"ContainerDied","Data":"63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7"} Nov 02 23:21:49.282021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.282012 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c1ba67ec5ceb4b42461d2a2c65df8188136c783f3b24376dca2307d88e50a7" Nov 02 23:21:49.283812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.283789 4494 generic.go:356] "Generic (PLEG): container finished" podID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" containerID="5f42f40ccabd03dc8f730e234816139364687046ddef07bd43230025c15e62c2" exitCode=0 Nov 02 23:21:49.283940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.283834 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" event={"ID":"880f056c-108b-4abc-a5a7-ac1c1bb98dd7","Type":"ContainerDied","Data":"5f42f40ccabd03dc8f730e234816139364687046ddef07bd43230025c15e62c2"} Nov 02 23:21:49.306118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.306089 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:21:49.368483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.368456 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:21:49.399056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.399024 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\") pod \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " Nov 02 23:21:49.399254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.399240 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\" (UniqueName: \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\") pod \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\" (UID: \"1d0e97d3-63b8-46b5-84cf-9be382585fcc\") " Nov 02 23:21:49.399405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.399383 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume" pod "1d0e97d3-63b8-46b5-84cf-9be382585fcc" (UID: "1d0e97d3-63b8-46b5-84cf-9be382585fcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:49.399660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.399644 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d0e97d3-63b8-46b5-84cf-9be382585fcc-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:49.401459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.401439 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6" pod "1d0e97d3-63b8-46b5-84cf-9be382585fcc" (UID: "1d0e97d3-63b8-46b5-84cf-9be382585fcc"). InnerVolumeSpecName "kube-api-access-k4cx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:49.529251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.513574 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\" (UniqueName: \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\") pod \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " Nov 02 23:21:49.529251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.513798 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\") pod \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\" (UID: \"880f056c-108b-4abc-a5a7-ac1c1bb98dd7\") " Nov 02 23:21:49.529251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.514395 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4cx6\" (UniqueName: \"kubernetes.io/projected/1d0e97d3-63b8-46b5-84cf-9be382585fcc-kube-api-access-k4cx6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:49.529251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.526092 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj" pod "880f056c-108b-4abc-a5a7-ac1c1bb98dd7" (UID: "880f056c-108b-4abc-a5a7-ac1c1bb98dd7"). InnerVolumeSpecName "kube-api-access-jzhjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:49.575203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.574466 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume" pod "880f056c-108b-4abc-a5a7-ac1c1bb98dd7" (UID: "880f056c-108b-4abc-a5a7-ac1c1bb98dd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:49.591758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.589118 4494 patch_prober.go:29] interesting pod/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:21:49.591758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.589193 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 23:21:49.618087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.618055 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhjj\" (UniqueName: \"kubernetes.io/projected/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-kube-api-access-jzhjj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:49.618378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:49.618357 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880f056c-108b-4abc-a5a7-ac1c1bb98dd7-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:50.292324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.292291 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" Nov 02 23:21:50.292528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.292325 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn" event={"ID":"880f056c-108b-4abc-a5a7-ac1c1bb98dd7","Type":"ContainerDied","Data":"d2077f35e7d87e7de47fb5fb4c914f59a3a5fe34c7c18c8ae4202b5e912145cf"} Nov 02 23:21:50.292528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.292385 4494 scope.go:122] "RemoveContainer" containerID="5f42f40ccabd03dc8f730e234816139364687046ddef07bd43230025c15e62c2" Nov 02 23:21:50.292528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.292478 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x" Nov 02 23:21:50.469502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.469468 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:21:50.478057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.477993 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-aa2454-7fd76c74c5-4tpgn"] Nov 02 23:21:50.487137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.487107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:50.594257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.593853 4494 patch_prober.go:29] interesting pod/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:21:50.594257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.593932 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 23:21:50.600219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.600190 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:21:50.605366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:50.605342 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-aa2454-859dcf9c5b-rcw9x"] Nov 02 23:21:51.586386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:51.586350 4494 patch_prober.go:29] interesting pod/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 23:21:51.586819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:51.586411 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 23:21:51.695003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:51.694965 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" path="/var/lib/kubelet/pods/1d0e97d3-63b8-46b5-84cf-9be382585fcc/volumes" Nov 02 23:21:51.695520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:51.695494 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" path="/var/lib/kubelet/pods/880f056c-108b-4abc-a5a7-ac1c1bb98dd7/volumes" Nov 02 23:21:52.189726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.189691 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:52.293201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.291286 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:21:52.336968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.336929 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:52.612221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.612118 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:21:52.614444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.614422 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:21:52.788195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.788140 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" podUID="6588ecac-78f7-49c3-941d-d6fa831b386b" containerName="payload-processing" containerID="cri-o://abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca" gracePeriod=25 Nov 02 23:21:52.881111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:52.880862 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" podUID="d0f671d2-fea5-4493-89c6-6042de61b05d" containerName="payload-pre-processing" containerID="cri-o://5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e" gracePeriod=25 Nov 02 23:21:53.170686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.170660 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:21:53.189353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.189330 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\") pod \"6588ecac-78f7-49c3-941d-d6fa831b386b\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " Nov 02 23:21:53.189481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.189466 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\" (UniqueName: \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\") pod \"6588ecac-78f7-49c3-941d-d6fa831b386b\" (UID: \"6588ecac-78f7-49c3-941d-d6fa831b386b\") " Nov 02 23:21:53.189703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.189680 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume" pod "6588ecac-78f7-49c3-941d-d6fa831b386b" (UID: "6588ecac-78f7-49c3-941d-d6fa831b386b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:53.189989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.189968 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6588ecac-78f7-49c3-941d-d6fa831b386b-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:53.191705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.191687 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl" pod "6588ecac-78f7-49c3-941d-d6fa831b386b" (UID: "6588ecac-78f7-49c3-941d-d6fa831b386b"). InnerVolumeSpecName "kube-api-access-bslkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:53.199441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.199421 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:21:53.291741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.291688 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\" (UniqueName: \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\") pod \"d0f671d2-fea5-4493-89c6-6042de61b05d\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " Nov 02 23:21:53.292788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.291946 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\") pod \"d0f671d2-fea5-4493-89c6-6042de61b05d\" (UID: \"d0f671d2-fea5-4493-89c6-6042de61b05d\") " Nov 02 23:21:53.292788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.292449 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslkl\" (UniqueName: \"kubernetes.io/projected/6588ecac-78f7-49c3-941d-d6fa831b386b-kube-api-access-bslkl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:53.292943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.292852 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume" pod "d0f671d2-fea5-4493-89c6-6042de61b05d" (UID: "d0f671d2-fea5-4493-89c6-6042de61b05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:53.294859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.294832 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw" pod "d0f671d2-fea5-4493-89c6-6042de61b05d" (UID: "d0f671d2-fea5-4493-89c6-6042de61b05d"). InnerVolumeSpecName "kube-api-access-4k4gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:53.315061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.314980 4494 generic.go:356] "Generic (PLEG): container finished" podID="d0f671d2-fea5-4493-89c6-6042de61b05d" containerID="5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e" exitCode=0 Nov 02 23:21:53.315061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.315045 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" Nov 02 23:21:53.315342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.315182 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" event={"ID":"d0f671d2-fea5-4493-89c6-6042de61b05d","Type":"ContainerDied","Data":"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e"} Nov 02 23:21:53.315342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.315265 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr" event={"ID":"d0f671d2-fea5-4493-89c6-6042de61b05d","Type":"ContainerDied","Data":"e16b70a081760229488cc7a6aec9e51e5826e506679146ed07fc5044616a74cc"} Nov 02 23:21:53.315342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.315299 4494 scope.go:122] "RemoveContainer" containerID="5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e" Nov 02 23:21:53.317580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.317458 4494 generic.go:356] "Generic (PLEG): container finished" podID="6588ecac-78f7-49c3-941d-d6fa831b386b" containerID="abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca" exitCode=0 Nov 02 23:21:53.318343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.318322 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" Nov 02 23:21:53.321971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.321268 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" event={"ID":"6588ecac-78f7-49c3-941d-d6fa831b386b","Type":"ContainerDied","Data":"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca"} Nov 02 23:21:53.321971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.321326 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r" event={"ID":"6588ecac-78f7-49c3-941d-d6fa831b386b","Type":"ContainerDied","Data":"24286e7fcdc86e806e8c4c6ab434d1442e2120394d307d35e3325095b7c74ef8"} Nov 02 23:21:53.327377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.327278 4494 scope.go:122] "RemoveContainer" containerID="5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e" Nov 02 23:21:53.327737 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:53.327715 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e\": container with ID starting with 5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e not found: ID does not exist" containerID="5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e" Nov 02 23:21:53.327829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.327744 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e"} err="failed to get container status \"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e\": rpc error: code = NotFound desc = could not find container \"5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e\": container with ID starting with 5fa314995c365ca454c1421b3b843d72cd58c9ba154e2a7d88f7ac5f368c933e not found: ID does not exist" Nov 02 23:21:53.327829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.327767 4494 scope.go:122] "RemoveContainer" containerID="abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca" Nov 02 23:21:53.337611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.337592 4494 scope.go:122] "RemoveContainer" containerID="abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca" Nov 02 23:21:53.337862 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:53.337844 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca\": container with ID starting with abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca not found: ID does not exist" containerID="abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca" Nov 02 23:21:53.337915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.337870 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca"} err="failed to get container status \"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca\": rpc error: code = NotFound desc = could not find container \"abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca\": container with ID starting with abed60ece5f955cfd8b1df7fd374ff92e8d64b9b6306e41ead1a77e9cb4807ca not found: ID does not exist" Nov 02 23:21:53.393785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.393711 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0f671d2-fea5-4493-89c6-6042de61b05d-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:53.393785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.393736 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k4gw\" (UniqueName: \"kubernetes.io/projected/d0f671d2-fea5-4493-89c6-6042de61b05d-kube-api-access-4k4gw\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:53.517558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.517527 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:21:53.527160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.527134 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee5b4-f786dbb9c-mb62r"] Nov 02 23:21:53.636257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.636224 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:21:53.644229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.644128 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee5b4-7dfdddf975-5nnvr"] Nov 02 23:21:53.695013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.694984 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6588ecac-78f7-49c3-941d-d6fa831b386b" path="/var/lib/kubelet/pods/6588ecac-78f7-49c3-941d-d6fa831b386b/volumes" Nov 02 23:21:53.695423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.695408 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f671d2-fea5-4493-89c6-6042de61b05d" path="/var/lib/kubelet/pods/d0f671d2-fea5-4493-89c6-6042de61b05d/volumes" Nov 02 23:21:53.704127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:53.704103 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:54.167835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:54.167799 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:21:54.168060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:54.168037 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" containerID="cri-o://6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63" gracePeriod=30 Nov 02 23:21:54.203201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:54.203122 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:54.677489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:54.677457 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:54.945971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:54.945872 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:55.908011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:55.907965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:57.389626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:57.389593 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:21:57.390074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:57.389668 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:21:59.432554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.432486 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:21:59.456818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.456789 4494 generic.go:356] "Generic (PLEG): container finished" podID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerID="6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63" exitCode=0 Nov 02 23:21:59.456951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.456840 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" Nov 02 23:21:59.456951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.456874 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerDied","Data":"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63"} Nov 02 23:21:59.457094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.456955 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9" event={"ID":"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f","Type":"ContainerDied","Data":"0d9596022c3531dfb99ea7cce0e126a93ef5cf8685a4fc115d27cc9b19e4e9c6"} Nov 02 23:21:59.457094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.456980 4494 scope.go:122] "RemoveContainer" containerID="6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63" Nov 02 23:21:59.466748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.466731 4494 scope.go:122] "RemoveContainer" containerID="2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f" Nov 02 23:21:59.475601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.475576 4494 scope.go:122] "RemoveContainer" containerID="6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63" Nov 02 23:21:59.475891 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:59.475863 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63\": container with ID starting with 6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63 not found: ID does not exist" containerID="6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63" Nov 02 23:21:59.475945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.475903 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63"} err="failed to get container status \"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63\": rpc error: code = NotFound desc = could not find container \"6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63\": container with ID starting with 6d5050f6702a6bd75257a80d04ff0872969735b3fd8d98ee7fc790b069b32a63 not found: ID does not exist" Nov 02 23:21:59.475945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.475921 4494 scope.go:122] "RemoveContainer" containerID="2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f" Nov 02 23:21:59.476228 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:21:59.476210 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f\": container with ID starting with 2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f not found: ID does not exist" containerID="2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f" Nov 02 23:21:59.476283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.476234 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f"} err="failed to get container status \"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f\": rpc error: code = NotFound desc = could not find container \"2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f\": container with ID starting with 2095011f68a111e277ced0805b6777add4846a64854d93c265448f59d93e0c8f not found: ID does not exist" Nov 02 23:21:59.544309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544267 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544339 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544456 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544521 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544580 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544611 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544638 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544730 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544843 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\") pod \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\" (UID: \"5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f\") " Nov 02 23:21:59.544894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.544849 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:59.545480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.545283 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.545480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.545311 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:59.545480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.545307 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:59.545480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.545396 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:59.546048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.546015 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:59.546746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.546726 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:59.546951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.546924 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "kube-api-access-7zt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:59.547278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.547254 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:21:59.547617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.547599 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token" pod "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" (UID: "5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:59.646856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646814 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zt47\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-kube-api-access-7zt47\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.646856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646844 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.646856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646855 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.647099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646869 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.647099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646877 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.647099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646885 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.647099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646893 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.647099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.646905 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:21:59.744448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.744416 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:21:59.875490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.875428 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:21:59.879024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:21:59.879000 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa2454-openshift-default-7f44f9b9b9-frfd9"] Nov 02 23:22:00.332672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:00.332644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:01.227958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.227928 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:01.437919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.437872 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:22:01.438613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438592 4494 cpu_manager.go:414] "containerMap: removing container" podUID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" containerName="payload-pre-processing" Nov 02 23:22:01.438613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438613 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" containerName="payload-pre-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438637 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438644 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438654 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6588ecac-78f7-49c3-941d-d6fa831b386b" containerName="payload-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438662 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6588ecac-78f7-49c3-941d-d6fa831b386b" containerName="payload-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438698 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" containerName="payload-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438705 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" containerName="payload-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438734 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d0f671d2-fea5-4493-89c6-6042de61b05d" containerName="payload-pre-processing" Nov 02 23:22:01.438794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438742 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f671d2-fea5-4493-89c6-6042de61b05d" containerName="payload-pre-processing" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438904 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f671d2-fea5-4493-89c6-6042de61b05d" containerName="payload-pre-processing" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438925 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="880f056c-108b-4abc-a5a7-ac1c1bb98dd7" containerName="payload-pre-processing" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438938 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438947 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6588ecac-78f7-49c3-941d-d6fa831b386b" containerName="payload-processing" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438963 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:01.439215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.438973 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0e97d3-63b8-46b5-84cf-9be382585fcc" containerName="payload-processing" Nov 02 23:22:01.443684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.443662 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.499497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.499412 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:22:01.546213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.546153 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b1c72\"" Nov 02 23:22:01.546213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.546197 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b1c72-dockercfg-gmpgh\"" Nov 02 23:22:01.574571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.574544 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.574744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.574725 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgmq\" (UniqueName: \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.676605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.676565 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgmq\" (UniqueName: \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.676795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.676739 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.677461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.677438 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.693871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.693841 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" path="/var/lib/kubelet/pods/5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f/volumes" Nov 02 23:22:01.738152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.738119 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgmq\" (UniqueName: \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\") pod \"payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:01.757258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:01.757161 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:02.106113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.106029 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:22:02.106676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.106663 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:02.106723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.106678 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9fb997-5eec-4d00-b7eb-0dfbdb3f147f" containerName="istio-proxy" Nov 02 23:22:02.110224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.110207 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.135576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.135547 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:22:02.136005 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:22:02.135983 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7fa98e_44fa_4baf_bc1a_3d886bd790e3.slice/crio-24456842b91e360ef78a4b9f01e440102affb715bec0bbae1db0d1738f9606ee WatchSource:0}: Error finding container 24456842b91e360ef78a4b9f01e440102affb715bec0bbae1db0d1738f9606ee: Status 404 returned error can't find the container with id 24456842b91e360ef78a4b9f01e440102affb715bec0bbae1db0d1738f9606ee Nov 02 23:22:02.168149 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.168120 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:22:02.183734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.183699 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6mb\" (UniqueName: \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.183875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.183745 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.285592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.285544 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6mb\" (UniqueName: \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.285592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.285601 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.286415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.286394 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.347845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.347804 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6mb\" (UniqueName: \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\") pod \"payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.424770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.424734 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:02.497210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.488761 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" event={"ID":"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3","Type":"ContainerStarted","Data":"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe"} Nov 02 23:22:02.497210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.488826 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" event={"ID":"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3","Type":"ContainerStarted","Data":"24456842b91e360ef78a4b9f01e440102affb715bec0bbae1db0d1738f9606ee"} Nov 02 23:22:02.497210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.488859 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:02.609136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.609076 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" podStartSLOduration=1.609063188 podStartE2EDuration="1.609063188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:02.606995929 +0000 UTC m=+2037.036865790" watchObservedRunningTime="2026-11-02 23:22:02.609063188 +0000 UTC m=+2037.038933442" Nov 02 23:22:02.831554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:02.824356 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:22:03.108440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:03.108343 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:03.496536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:03.496495 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" event={"ID":"c341b213-a714-438f-8c3f-94eb28234934","Type":"ContainerStarted","Data":"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a"} Nov 02 23:22:03.497047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:03.496550 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" event={"ID":"c341b213-a714-438f-8c3f-94eb28234934","Type":"ContainerStarted","Data":"cb0d16ca13bc0722f0d445e3256a8683905cd743b4a68522095a1fd39436560e"} Nov 02 23:22:03.497047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:03.496741 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:03.614572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:03.614520 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" podStartSLOduration=1.6145066030000002 podStartE2EDuration="1.614506603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:03.61365379 +0000 UTC m=+2038.043523653" watchObservedRunningTime="2026-11-02 23:22:03.614506603 +0000 UTC m=+2038.044376515" Nov 02 23:22:06.927833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:06.927799 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:08.861594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:08.861531 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:11.516871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:11.516836 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:12.856049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:12.856013 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:13.499143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:13.499113 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:14.446124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:14.446094 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:14.508278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:14.508154 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:14.620608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:14.620581 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:22:14.676203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:14.676142 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:22:14.928825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:14.928797 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:15.072931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:15.072900 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:17.561698 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:17.561659 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:22:18.063402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:18.063357 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:19.621018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:19.620979 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" podUID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" containerName="payload-processing" containerID="cri-o://b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7" gracePeriod=25 Nov 02 23:22:19.678868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:19.678832 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" podUID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" containerName="payload-pre-processing" containerID="cri-o://e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f" gracePeriod=25 Nov 02 23:22:19.949716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:19.949692 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:22:19.987066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:19.987042 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:22:20.023080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023055 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\") pod \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " Nov 02 23:22:20.023256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023100 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\") pod \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " Nov 02 23:22:20.023256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023216 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\" (UniqueName: \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\") pod \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\" (UID: \"ada5245d-2663-4686-a035-e5fbd7bf4e8f\") " Nov 02 23:22:20.023371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023292 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\" (UniqueName: \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\") pod \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\" (UID: \"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed\") " Nov 02 23:22:20.023568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023533 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume" pod "ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" (UID: "ef04ff2b-0aa6-4d5d-aed6-4520c75574ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:20.023697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023589 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume" pod "ada5245d-2663-4686-a035-e5fbd7bf4e8f" (UID: "ada5245d-2663-4686-a035-e5fbd7bf4e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:20.023931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023906 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.024048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.023934 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada5245d-2663-4686-a035-e5fbd7bf4e8f-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.025754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.025731 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz" pod "ada5245d-2663-4686-a035-e5fbd7bf4e8f" (UID: "ada5245d-2663-4686-a035-e5fbd7bf4e8f"). InnerVolumeSpecName "kube-api-access-r8gcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:20.025833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.025785 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd" pod "ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" (UID: "ef04ff2b-0aa6-4d5d-aed6-4520c75574ed"). InnerVolumeSpecName "kube-api-access-rwgrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:20.125775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.125743 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gcz\" (UniqueName: \"kubernetes.io/projected/ada5245d-2663-4686-a035-e5fbd7bf4e8f-kube-api-access-r8gcz\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.125775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.125770 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwgrd\" (UniqueName: \"kubernetes.io/projected/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed-kube-api-access-rwgrd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:20.617420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.617382 4494 generic.go:356] "Generic (PLEG): container finished" podID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" containerID="b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7" exitCode=0 Nov 02 23:22:20.617420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.617424 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" Nov 02 23:22:20.617696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.617478 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" event={"ID":"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed","Type":"ContainerDied","Data":"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7"} Nov 02 23:22:20.617696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.617524 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg" event={"ID":"ef04ff2b-0aa6-4d5d-aed6-4520c75574ed","Type":"ContainerDied","Data":"c640de0cbe182c9afa2e24d2a0a73ef84a539506780c35ed5ee093ba5426f0d0"} Nov 02 23:22:20.617696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.617547 4494 scope.go:122] "RemoveContainer" containerID="b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7" Nov 02 23:22:20.619065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.619043 4494 generic.go:356] "Generic (PLEG): container finished" podID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" containerID="e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f" exitCode=0 Nov 02 23:22:20.619216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.619095 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" Nov 02 23:22:20.619216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.619127 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" event={"ID":"ada5245d-2663-4686-a035-e5fbd7bf4e8f","Type":"ContainerDied","Data":"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f"} Nov 02 23:22:20.619216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.619200 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r" event={"ID":"ada5245d-2663-4686-a035-e5fbd7bf4e8f","Type":"ContainerDied","Data":"80e2ac6d01a08dc50c7ea286f6a07c356cddff9494032a143296723086dfc411"} Nov 02 23:22:20.629531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.629353 4494 scope.go:122] "RemoveContainer" containerID="b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7" Nov 02 23:22:20.629784 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:20.629650 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7\": container with ID starting with b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7 not found: ID does not exist" containerID="b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7" Nov 02 23:22:20.629784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.629676 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7"} err="failed to get container status \"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7\": rpc error: code = NotFound desc = could not find container \"b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7\": container with ID starting with b7bfc825e4e28c49a1035cdac3a59db49d97ae6eaf2f63f2689d2772bf256ba7 not found: ID does not exist" Nov 02 23:22:20.629784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.629693 4494 scope.go:122] "RemoveContainer" containerID="e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f" Nov 02 23:22:20.647352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.647335 4494 scope.go:122] "RemoveContainer" containerID="e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f" Nov 02 23:22:20.647615 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:20.647594 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f\": container with ID starting with e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f not found: ID does not exist" containerID="e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f" Nov 02 23:22:20.647679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.647629 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f"} err="failed to get container status \"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f\": rpc error: code = NotFound desc = could not find container \"e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f\": container with ID starting with e8b40d84cd0971908a9427b7d928e60e663977826f52e25539b49f8ca2c9559f not found: ID does not exist" Nov 02 23:22:20.784673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.784642 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:22:20.788592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.788566 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-7c4c875799-w5g7r"] Nov 02 23:22:20.900055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.899950 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:22:20.903360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:20.903342 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-6866c7bff5-srjxg"] Nov 02 23:22:21.693904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:21.693864 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" path="/var/lib/kubelet/pods/ada5245d-2663-4686-a035-e5fbd7bf4e8f/volumes" Nov 02 23:22:21.694373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:21.694357 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" path="/var/lib/kubelet/pods/ef04ff2b-0aa6-4d5d-aed6-4520c75574ed/volumes" Nov 02 23:22:22.960127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:22.960089 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:23.644822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:23.644787 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:22:27.996271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:27.996236 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:32.233094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.233036 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:22:32.252539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.252492 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" podUID="b0286822-fd1a-4270-8d92-77e72cec367e" containerName="maas-api" containerID="cri-o://4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748" gracePeriod=30 Nov 02 23:22:32.322543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.322402 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:32.614776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.614748 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:22:32.684765 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.684720 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:22:32.684971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.684954 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" podUID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" containerName="payload-processing" containerID="cri-o://4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5" gracePeriod=30 Nov 02 23:22:32.721490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.721455 4494 generic.go:356] "Generic (PLEG): container finished" podID="b0286822-fd1a-4270-8d92-77e72cec367e" containerID="4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748" exitCode=0 Nov 02 23:22:32.721686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.721514 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" Nov 02 23:22:32.721686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.721564 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" event={"ID":"b0286822-fd1a-4270-8d92-77e72cec367e","Type":"ContainerDied","Data":"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748"} Nov 02 23:22:32.721686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.721655 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm" event={"ID":"b0286822-fd1a-4270-8d92-77e72cec367e","Type":"ContainerDied","Data":"23a63f4f273c642b711aa60f4b3d086612d103a9ccc1e33d40a3c3abf61b42c3"} Nov 02 23:22:32.721863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.721696 4494 scope.go:122] "RemoveContainer" containerID="4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748" Nov 02 23:22:32.728360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.728338 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\") pod \"b0286822-fd1a-4270-8d92-77e72cec367e\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " Nov 02 23:22:32.728524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.728510 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") pod \"b0286822-fd1a-4270-8d92-77e72cec367e\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " Nov 02 23:22:32.728572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.728533 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\") pod \"b0286822-fd1a-4270-8d92-77e72cec367e\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " Nov 02 23:22:32.728668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.728651 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\" (UniqueName: \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\") pod \"b0286822-fd1a-4270-8d92-77e72cec367e\" (UID: \"b0286822-fd1a-4270-8d92-77e72cec367e\") " Nov 02 23:22:32.728668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.728657 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca" pod "b0286822-fd1a-4270-8d92-77e72cec367e" (UID: "b0286822-fd1a-4270-8d92-77e72cec367e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:32.729282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.729252 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0286822-fd1a-4270-8d92-77e72cec367e-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:32.739270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.739214 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls" pod "b0286822-fd1a-4270-8d92-77e72cec367e" (UID: "b0286822-fd1a-4270-8d92-77e72cec367e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:32.740093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.740061 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs" pod "b0286822-fd1a-4270-8d92-77e72cec367e" (UID: "b0286822-fd1a-4270-8d92-77e72cec367e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:32.743796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.743760 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:22:32.744264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.744236 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" podUID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" containerName="payload-pre-processing" containerID="cri-o://680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa" gracePeriod=30 Nov 02 23:22:32.750555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.750523 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh" pod "b0286822-fd1a-4270-8d92-77e72cec367e" (UID: "b0286822-fd1a-4270-8d92-77e72cec367e"). InnerVolumeSpecName "kube-api-access-7wbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:32.814664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.814634 4494 scope.go:122] "RemoveContainer" containerID="4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748" Nov 02 23:22:32.815927 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:32.815893 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748\": container with ID starting with 4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748 not found: ID does not exist" containerID="4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748" Nov 02 23:22:32.816038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.815938 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748"} err="failed to get container status \"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748\": rpc error: code = NotFound desc = could not find container \"4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748\": container with ID starting with 4551ab3e0c355a1ff28b33fc0c777e48728a4c2557b51cd342fa7411ab3e3748 not found: ID does not exist" Nov 02 23:22:32.830769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.830734 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:32.830769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.830770 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0286822-fd1a-4270-8d92-77e72cec367e-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:32.831007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.830785 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wbzh\" (UniqueName: \"kubernetes.io/projected/b0286822-fd1a-4270-8d92-77e72cec367e-kube-api-access-7wbzh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:32.846130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.846051 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:32.992883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.992847 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:32.996247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:32.996224 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:22:33.046001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.045975 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:22:33.133519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.133488 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\") pod \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " Nov 02 23:22:33.133719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.133568 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\" (UniqueName: \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\") pod \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " Nov 02 23:22:33.133719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.133630 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\" (UniqueName: \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\") pod \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\" (UID: \"aa5b29cf-27e4-4b89-b5c0-07a8055534ae\") " Nov 02 23:22:33.133835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.133761 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\") pod \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\" (UID: \"ba90937f-b697-46f8-9709-d8a4d69c5c6b\") " Nov 02 23:22:33.133894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.133860 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume" pod "aa5b29cf-27e4-4b89-b5c0-07a8055534ae" (UID: "aa5b29cf-27e4-4b89-b5c0-07a8055534ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:33.134211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.134194 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:33.134294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.134225 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume" pod "ba90937f-b697-46f8-9709-d8a4d69c5c6b" (UID: "ba90937f-b697-46f8-9709-d8a4d69c5c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:33.136521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.136499 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k" pod "ba90937f-b697-46f8-9709-d8a4d69c5c6b" (UID: "ba90937f-b697-46f8-9709-d8a4d69c5c6b"). InnerVolumeSpecName "kube-api-access-khh9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:33.136597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.136531 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs" pod "aa5b29cf-27e4-4b89-b5c0-07a8055534ae" (UID: "aa5b29cf-27e4-4b89-b5c0-07a8055534ae"). InnerVolumeSpecName "kube-api-access-6nfrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:33.240311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.235865 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khh9k\" (UniqueName: \"kubernetes.io/projected/ba90937f-b697-46f8-9709-d8a4d69c5c6b-kube-api-access-khh9k\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:33.240311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.235905 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfrs\" (UniqueName: \"kubernetes.io/projected/aa5b29cf-27e4-4b89-b5c0-07a8055534ae-kube-api-access-6nfrs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:33.240311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.235919 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba90937f-b697-46f8-9709-d8a4d69c5c6b-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:33.278794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.278603 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:22:33.324011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.323918 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b4bf70-6454cf5b5-6dbpm"] Nov 02 23:22:33.694904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.694869 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0286822-fd1a-4270-8d92-77e72cec367e" path="/var/lib/kubelet/pods/b0286822-fd1a-4270-8d92-77e72cec367e/volumes" Nov 02 23:22:33.731996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.731962 4494 generic.go:356] "Generic (PLEG): container finished" podID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" containerID="4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5" exitCode=0 Nov 02 23:22:33.732376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.732049 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" event={"ID":"ba90937f-b697-46f8-9709-d8a4d69c5c6b","Type":"ContainerDied","Data":"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5"} Nov 02 23:22:33.732376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.732069 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" Nov 02 23:22:33.732376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.732127 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4" event={"ID":"ba90937f-b697-46f8-9709-d8a4d69c5c6b","Type":"ContainerDied","Data":"085c342a3ffbd57e41ae3d86741bc4fda4b6504ca44150c29744ff4cace15127"} Nov 02 23:22:33.732376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.732161 4494 scope.go:122] "RemoveContainer" containerID="4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5" Nov 02 23:22:33.733993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.733964 4494 generic.go:356] "Generic (PLEG): container finished" podID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" containerID="680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa" exitCode=0 Nov 02 23:22:33.734147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.734075 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" Nov 02 23:22:33.734316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.734076 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" event={"ID":"aa5b29cf-27e4-4b89-b5c0-07a8055534ae","Type":"ContainerDied","Data":"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa"} Nov 02 23:22:33.734477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.734452 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn" event={"ID":"aa5b29cf-27e4-4b89-b5c0-07a8055534ae","Type":"ContainerDied","Data":"04acf58faf307e061be26a182e860150525725f2b2a9c016053996d263ccf466"} Nov 02 23:22:33.742290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.742269 4494 scope.go:122] "RemoveContainer" containerID="4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5" Nov 02 23:22:33.742568 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:33.742540 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5\": container with ID starting with 4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5 not found: ID does not exist" containerID="4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5" Nov 02 23:22:33.742638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.742576 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5"} err="failed to get container status \"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5\": rpc error: code = NotFound desc = could not find container \"4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5\": container with ID starting with 4145594bfc5f9578ddc3a2eac3123b6e7c5a9b37f64dd83be5dd8a1f602cd2f5 not found: ID does not exist" Nov 02 23:22:33.742638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.742599 4494 scope.go:122] "RemoveContainer" containerID="680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa" Nov 02 23:22:33.751763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.751742 4494 scope.go:122] "RemoveContainer" containerID="680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa" Nov 02 23:22:33.752110 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:33.752082 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa\": container with ID starting with 680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa not found: ID does not exist" containerID="680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa" Nov 02 23:22:33.752212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.752121 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa"} err="failed to get container status \"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa\": rpc error: code = NotFound desc = could not find container \"680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa\": container with ID starting with 680281a9bf8e5d3c74f06030c981a6fc77186720d0008f1165b53e84e78b02fa not found: ID does not exist" Nov 02 23:22:33.900336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.900191 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:22:33.906426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:33.906388 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4bf70-5997bd4c56-7ztw4"] Nov 02 23:22:34.014307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:34.014220 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:22:34.020793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:34.020762 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf70-7bf6dc6747-7m4vn"] Nov 02 23:22:34.405043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:34.404957 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:35.693478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.693447 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" path="/var/lib/kubelet/pods/aa5b29cf-27e4-4b89-b5c0-07a8055534ae/volumes" Nov 02 23:22:35.693924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.693821 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" path="/var/lib/kubelet/pods/ba90937f-b697-46f8-9709-d8a4d69c5c6b/volumes" Nov 02 23:22:35.745020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.744989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:35.750627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.750594 4494 generic.go:356] "Generic (PLEG): container finished" podID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerID="06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005" exitCode=0 Nov 02 23:22:35.750839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.750816 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerDied","Data":"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005"} Nov 02 23:22:35.751387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.751362 4494 scope.go:122] "RemoveContainer" containerID="06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005" Nov 02 23:22:35.775801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.772835 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:35.775801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.773069 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:35.775801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.773120 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:35.942097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:35.942062 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:36.397438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:36.397386 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:36.774303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:36.774266 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerStarted","Data":"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6"} Nov 02 23:22:37.091726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.091645 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:22:37.091890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.091846 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 23:22:37.091941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.091908 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 23:22:37.183342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.183291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:37.377956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.377847 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:37.670286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.667239 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:22:37.670286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.667511 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" containerID="cri-o://b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6" gracePeriod=30 Nov 02 23:22:37.678379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.678333 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:37.761026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.760988 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:22:37.761740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761721 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" containerName="payload-pre-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761742 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" containerName="payload-pre-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761762 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" containerName="payload-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761768 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" containerName="payload-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761781 4494 cpu_manager.go:414] "containerMap: removing container" podUID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" containerName="payload-pre-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761786 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" containerName="payload-pre-processing" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761800 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b0286822-fd1a-4270-8d92-77e72cec367e" containerName="maas-api" Nov 02 23:22:37.761812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761805 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0286822-fd1a-4270-8d92-77e72cec367e" containerName="maas-api" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761818 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" containerName="payload-processing" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761824 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" containerName="payload-processing" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761956 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5b29cf-27e4-4b89-b5c0-07a8055534ae" containerName="payload-pre-processing" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761970 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0286822-fd1a-4270-8d92-77e72cec367e" containerName="maas-api" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.761985 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef04ff2b-0aa6-4d5d-aed6-4520c75574ed" containerName="payload-processing" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.762001 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada5245d-2663-4686-a035-e5fbd7bf4e8f" containerName="payload-pre-processing" Nov 02 23:22:37.762065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.762008 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba90937f-b697-46f8-9709-d8a4d69c5c6b" containerName="payload-processing" Nov 02 23:22:37.765810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.765791 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.824971 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825059 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825129 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825216 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825275 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825376 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn5m\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825411 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825465 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.825830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.825499 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:37.939673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.913643 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e875bc-openshift-default-dockercfg-86s9r\"" Nov 02 23:22:37.939673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:37.923992 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.004823 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.004880 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.004940 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005002 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005076 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005141 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005218 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005330 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn5m\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.008747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.005364 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.023203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.020726 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.023477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.021156 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.025463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.025436 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.026386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.026358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.026505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.026482 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.027201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.027151 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.027663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.027642 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.064122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.064078 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:38.087703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.087675 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.091941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.091912 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn5m\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\") pod \"e2e-api-b-e875bc-openshift-default-595c678ff-kttqv\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.092084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.091914 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 23:22:38.092084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.092061 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 23:22:38.212873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.212794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:38.301271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.301242 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:38.417593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.417499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:38.763122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.763098 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:22:38.763835 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:22:38.763802 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38368803_a2bb_44c3_b4bd_e90f86f56b8a.slice/crio-6c5c7f593d26d8aec661f4fa7abf07e2286c1282989ae80fb8ba624c6a55cb5a WatchSource:0}: Error finding container 6c5c7f593d26d8aec661f4fa7abf07e2286c1282989ae80fb8ba624c6a55cb5a: Status 404 returned error can't find the container with id 6c5c7f593d26d8aec661f4fa7abf07e2286c1282989ae80fb8ba624c6a55cb5a Nov 02 23:22:38.766513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.766480 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:38.766612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.766546 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:38.766612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.766596 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:38.927190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.927132 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerStarted","Data":"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b"} Nov 02 23:22:38.927692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:38.927234 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerStarted","Data":"6c5c7f593d26d8aec661f4fa7abf07e2286c1282989ae80fb8ba624c6a55cb5a"} Nov 02 23:22:39.092525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:39.092389 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 23:22:39.092525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:39.092440 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 23:22:39.302308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:39.302275 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:39.303821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:39.303795 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:22:39.303936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:39.303839 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:22:40.092021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.091989 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 23:22:40.092521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.092046 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 23:22:40.306576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.306548 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:40.455974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.455918 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podStartSLOduration=3.4559016209999998 podStartE2EDuration="3.455901621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:39.05937007 +0000 UTC m=+2073.489239937" watchObservedRunningTime="2026-11-02 23:22:40.455901621 +0000 UTC m=+2074.885771481" Nov 02 23:22:40.960846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.960807 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:40.961999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:40.961980 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:22:41.095670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:41.095575 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 23:22:41.096052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:41.095822 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 23:22:41.755856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:41.755811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:41.914013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:41.913974 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:42.083275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.083153 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:42.186907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.186877 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:22:42.187634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.187593 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:22:42.188800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.188779 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:22:42.247463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.247430 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:22:42.247612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.247488 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:22:42.942989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.942931 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:22:42.977323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.977291 4494 generic.go:356] "Generic (PLEG): container finished" podID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerID="b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6" exitCode=0 Nov 02 23:22:42.977492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.977348 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" Nov 02 23:22:42.977492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.977371 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerDied","Data":"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6"} Nov 02 23:22:42.977492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.977425 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt" event={"ID":"c2b02c19-bd94-436d-90b6-4a03f095856c","Type":"ContainerDied","Data":"2eb1952c95fccb773d481b752c436879a526d090b234aaf6db4e59e2af5a47b6"} Nov 02 23:22:42.977492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.977452 4494 scope.go:122] "RemoveContainer" containerID="b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6" Nov 02 23:22:42.988365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:42.988347 4494 scope.go:122] "RemoveContainer" containerID="4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd" Nov 02 23:22:43.000564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.000543 4494 scope.go:122] "RemoveContainer" containerID="b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6" Nov 02 23:22:43.001003 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:43.000961 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6\": container with ID starting with b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6 not found: ID does not exist" containerID="b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6" Nov 02 23:22:43.001096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.001013 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6"} err="failed to get container status \"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6\": rpc error: code = NotFound desc = could not find container \"b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6\": container with ID starting with b7b3c506f98efd6cf6902f546164ee39966b56ecddd720ef21a58010eed76df6 not found: ID does not exist" Nov 02 23:22:43.001096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.001036 4494 scope.go:122] "RemoveContainer" containerID="4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd" Nov 02 23:22:43.001366 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:43.001340 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd\": container with ID starting with 4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd not found: ID does not exist" containerID="4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd" Nov 02 23:22:43.001426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.001375 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd"} err="failed to get container status \"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd\": rpc error: code = NotFound desc = could not find container \"4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd\": container with ID starting with 4581ba51ff9d9c6373e2bac0c1a450d8aa98de493d4dd3193abf9d39f77876cd not found: ID does not exist" Nov 02 23:22:43.092848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.092820 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.092931 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.092990 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093057 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093100 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093144 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093207 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093257 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.093397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093361 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\") pod \"c2b02c19-bd94-436d-90b6-4a03f095856c\" (UID: \"c2b02c19-bd94-436d-90b6-4a03f095856c\") " Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093536 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093546 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093781 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093760 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.093916 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.094457 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.094478 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b02c19-bd94-436d-90b6-4a03f095856c-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.094492 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.094505 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.095279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.094518 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.095981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.095958 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:43.096373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.096344 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:43.096503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.096467 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:43.096590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.096508 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj" pod "c2b02c19-bd94-436d-90b6-4a03f095856c" (UID: "c2b02c19-bd94-436d-90b6-4a03f095856c"). InnerVolumeSpecName "kube-api-access-vq9cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:43.196298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.196213 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.196298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.196242 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.196298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.196252 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b02c19-bd94-436d-90b6-4a03f095856c-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.196298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.196261 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9cj\" (UniqueName: \"kubernetes.io/projected/c2b02c19-bd94-436d-90b6-4a03f095856c-kube-api-access-vq9cj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:43.400935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.400900 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:22:43.404890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.404870 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf70-openshift-default-5c769989d9-ggctt"] Nov 02 23:22:43.694344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:43.694311 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" path="/var/lib/kubelet/pods/c2b02c19-bd94-436d-90b6-4a03f095856c/volumes" Nov 02 23:22:44.583815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:44.583786 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:45.054346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.054309 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:45.234009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.232603 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:45.627416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.627382 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:22:45.629078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.628931 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:45.629281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.629261 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:45.629573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.629559 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:45.629677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.629666 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:45.646461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.646428 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.686324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.686288 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:22:45.729668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.729631 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e875bc\"" Nov 02 23:22:45.729850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.729679 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e875bc-dockercfg-m8pl8\"" Nov 02 23:22:45.745268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.745237 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.745376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.745281 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls47b\" (UniqueName: \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.846148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.846113 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.846335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.846167 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls47b\" (UniqueName: \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.846865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.846846 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:45.907860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:45.907791 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls47b\" (UniqueName: \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\") pod \"payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:46.008473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.008436 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:46.385106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.385074 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:22:46.386078 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:22:46.386046 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a696e11_4df9_414e_b33b_075ec80ae374.slice/crio-964e9ea6c44f863ad42ecd56a6ab63e5fc5bcae91afe7c80c1c8245419b6ad49 WatchSource:0}: Error finding container 964e9ea6c44f863ad42ecd56a6ab63e5fc5bcae91afe7c80c1c8245419b6ad49: Status 404 returned error can't find the container with id 964e9ea6c44f863ad42ecd56a6ab63e5fc5bcae91afe7c80c1c8245419b6ad49 Nov 02 23:22:46.439424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.439391 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:22:46.439650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.439626 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" podUID="a58cd826-b563-46db-9e66-857898fa9666" containerName="maas-api" containerID="cri-o://bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe" gracePeriod=30 Nov 02 23:22:46.539930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.539830 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:46.756866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.756839 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:22:46.859055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.859018 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\" (UniqueName: \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\") pod \"a58cd826-b563-46db-9e66-857898fa9666\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " Nov 02 23:22:46.859323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.859105 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") pod \"a58cd826-b563-46db-9e66-857898fa9666\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " Nov 02 23:22:46.859323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.859235 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\") pod \"a58cd826-b563-46db-9e66-857898fa9666\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " Nov 02 23:22:46.859323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.859277 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\") pod \"a58cd826-b563-46db-9e66-857898fa9666\" (UID: \"a58cd826-b563-46db-9e66-857898fa9666\") " Nov 02 23:22:46.859682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.859660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca" pod "a58cd826-b563-46db-9e66-857898fa9666" (UID: "a58cd826-b563-46db-9e66-857898fa9666"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:46.861365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.861333 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls" pod "a58cd826-b563-46db-9e66-857898fa9666" (UID: "a58cd826-b563-46db-9e66-857898fa9666"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:46.861473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.861405 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls" pod "a58cd826-b563-46db-9e66-857898fa9666" (UID: "a58cd826-b563-46db-9e66-857898fa9666"). InnerVolumeSpecName "kube-api-access-v7nls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:46.861473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.861411 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs" pod "a58cd826-b563-46db-9e66-857898fa9666" (UID: "a58cd826-b563-46db-9e66-857898fa9666"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:46.961146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.961115 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:46.961146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.961142 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58cd826-b563-46db-9e66-857898fa9666-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:46.961381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.961153 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58cd826-b563-46db-9e66-857898fa9666-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:46.961381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:46.961164 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nls\" (UniqueName: \"kubernetes.io/projected/a58cd826-b563-46db-9e66-857898fa9666-kube-api-access-v7nls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:47.007218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.007121 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" event={"ID":"2a696e11-4df9-414e-b33b-075ec80ae374","Type":"ContainerStarted","Data":"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45"} Nov 02 23:22:47.007381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.007224 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:47.007381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.007255 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" event={"ID":"2a696e11-4df9-414e-b33b-075ec80ae374","Type":"ContainerStarted","Data":"964e9ea6c44f863ad42ecd56a6ab63e5fc5bcae91afe7c80c1c8245419b6ad49"} Nov 02 23:22:47.008688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.008655 4494 generic.go:356] "Generic (PLEG): container finished" podID="a58cd826-b563-46db-9e66-857898fa9666" containerID="bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe" exitCode=0 Nov 02 23:22:47.008819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.008696 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" event={"ID":"a58cd826-b563-46db-9e66-857898fa9666","Type":"ContainerDied","Data":"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe"} Nov 02 23:22:47.008819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.008700 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" Nov 02 23:22:47.008819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.008727 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h" event={"ID":"a58cd826-b563-46db-9e66-857898fa9666","Type":"ContainerDied","Data":"8b22af4063eb6aeb1773424c47e2de664fbb3ee4d7aa9d7e3cff85bff9ce6540"} Nov 02 23:22:47.008819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.008751 4494 scope.go:122] "RemoveContainer" containerID="bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe" Nov 02 23:22:47.022009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.021747 4494 scope.go:122] "RemoveContainer" containerID="bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe" Nov 02 23:22:47.022081 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:47.022016 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe\": container with ID starting with bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe not found: ID does not exist" containerID="bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe" Nov 02 23:22:47.022081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.022039 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe"} err="failed to get container status \"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe\": rpc error: code = NotFound desc = could not find container \"bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe\": container with ID starting with bff1b29c989ae6d6cd4b6eafd52b13dea050e311b8b3a6b756c3144c9f4973fe not found: ID does not exist" Nov 02 23:22:47.128369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.128314 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" podStartSLOduration=2.128298554 podStartE2EDuration="2.128298554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:47.126044882 +0000 UTC m=+2081.555914744" watchObservedRunningTime="2026-11-02 23:22:47.128298554 +0000 UTC m=+2081.558168415" Nov 02 23:22:47.267822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.264243 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:22:47.269244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.268905 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee5b4-5775f556fb-mq97h"] Nov 02 23:22:47.630884 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:47.630789 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:22:47.694995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:47.694962 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58cd826-b563-46db-9e66-857898fa9666" path="/var/lib/kubelet/pods/a58cd826-b563-46db-9e66-857898fa9666/volumes" Nov 02 23:22:48.565087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:48.565056 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:50.177991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:50.177956 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:50.323629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:50.323582 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:51.824394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:51.824358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:52.317053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:52.317024 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:22:52.317310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:52.317287 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" containerID="cri-o://58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818" gracePeriod=30 Nov 02 23:22:52.328491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:52.328462 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:53.767547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:53.767519 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:54.075937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.075861 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:54.260873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.260837 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:22:54.261635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.261605 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a58cd826-b563-46db-9e66-857898fa9666" containerName="maas-api" Nov 02 23:22:54.261635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.261627 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58cd826-b563-46db-9e66-857898fa9666" containerName="maas-api" Nov 02 23:22:54.261837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.261649 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:54.261837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.261656 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b02c19-bd94-436d-90b6-4a03f095856c" containerName="istio-proxy" Nov 02 23:22:54.261837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.261786 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58cd826-b563-46db-9e66-857898fa9666" containerName="maas-api" Nov 02 23:22:54.266425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.266396 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.324782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.324749 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-03fe6e-openshift-default-dockercfg-9nhhc\"" Nov 02 23:22:54.370114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.370011 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.395742 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.395816 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396054 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396105 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396134 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396202 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmfp\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396277 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.396309 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.372010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:54.397196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.384638 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:22:54.498456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498414 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498473 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498502 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498534 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmfp\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498593 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498623 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498679 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498729 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.498929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.498784 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.499402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.499378 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.499593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.499574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.500118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.500053 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.500558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.500505 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.501025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.501001 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.504079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.504063 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.506124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.506100 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.564218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.562645 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.567438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.567401 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmfp\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\") pod \"e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.581927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.581897 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:54.787571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:54.787526 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:55.011251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:55.011226 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:22:55.012367 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:22:55.012343 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3124d74_70b6_4a98_a039_d0a9a1c5c3f4.slice/crio-a6d056c2916b6e98cfec3fe105b093669d252565ce33101b5e8e6e810c032f91 WatchSource:0}: Error finding container a6d056c2916b6e98cfec3fe105b093669d252565ce33101b5e8e6e810c032f91: Status 404 returned error can't find the container with id a6d056c2916b6e98cfec3fe105b093669d252565ce33101b5e8e6e810c032f91 Nov 02 23:22:55.014891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:55.014861 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:55.014967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:55.014927 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:55.014967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:55.014957 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:22:55.064132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:55.064092 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerStarted","Data":"a6d056c2916b6e98cfec3fe105b093669d252565ce33101b5e8e6e810c032f91"} Nov 02 23:22:56.071716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:56.071680 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerStarted","Data":"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486"} Nov 02 23:22:56.194819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:56.194768 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podStartSLOduration=2.194753955 podStartE2EDuration="2.194753955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:56.192941547 +0000 UTC m=+2090.622811408" watchObservedRunningTime="2026-11-02 23:22:56.194753955 +0000 UTC m=+2090.624623816" Nov 02 23:22:56.582408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:56.582377 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:56.588395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:56.588370 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:57.077882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.077854 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:57.078954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.078935 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:22:57.593022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.592958 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:22:57.741900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.741863 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.741948 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742003 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742033 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742070 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742125 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742198 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742257 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742324 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\") pod \"f4202e77-afe6-4be4-91ae-7de75abeaadb\" (UID: \"f4202e77-afe6-4be4-91ae-7de75abeaadb\") " Nov 02 23:22:57.742633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742429 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:57.742633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742615 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:57.742830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.742804 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:57.743100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.743076 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:57.743418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.743396 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.743498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.743422 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4202e77-afe6-4be4-91ae-7de75abeaadb-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.743498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.743439 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.743661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.743639 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:57.744945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.744914 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "kube-api-access-svcc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:57.745056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.744982 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:57.745122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.745087 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:57.745276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.745258 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy" pod "f4202e77-afe6-4be4-91ae-7de75abeaadb" (UID: "f4202e77-afe6-4be4-91ae-7de75abeaadb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844564 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844594 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svcc4\" (UniqueName: \"kubernetes.io/projected/f4202e77-afe6-4be4-91ae-7de75abeaadb-kube-api-access-svcc4\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844605 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844614 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844622 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4202e77-afe6-4be4-91ae-7de75abeaadb-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:57.844644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:57.844630 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4202e77-afe6-4be4-91ae-7de75abeaadb-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:58.019989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.019961 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:22:58.086617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.086581 4494 generic.go:356] "Generic (PLEG): container finished" podID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerID="58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818" exitCode=0 Nov 02 23:22:58.087098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.086646 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" Nov 02 23:22:58.087098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.086656 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerDied","Data":"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818"} Nov 02 23:22:58.087098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.086716 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5" event={"ID":"f4202e77-afe6-4be4-91ae-7de75abeaadb","Type":"ContainerDied","Data":"8de68a4c2599dfd1d54c65e0efd455c1de0f9d0950fc4404cdca4272a82741b4"} Nov 02 23:22:58.087098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.086747 4494 scope.go:122] "RemoveContainer" containerID="58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818" Nov 02 23:22:58.098225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.098204 4494 scope.go:122] "RemoveContainer" containerID="84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123" Nov 02 23:22:58.107569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.107550 4494 scope.go:122] "RemoveContainer" containerID="58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818" Nov 02 23:22:58.107886 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:58.107864 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818\": container with ID starting with 58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818 not found: ID does not exist" containerID="58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818" Nov 02 23:22:58.107937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.107898 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818"} err="failed to get container status \"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818\": rpc error: code = NotFound desc = could not find container \"58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818\": container with ID starting with 58123ab86d4c8331cb7db485529df2d9d3c5f7a885d11f17cfdd41ee58c5c818 not found: ID does not exist" Nov 02 23:22:58.107937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.107918 4494 scope.go:122] "RemoveContainer" containerID="84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123" Nov 02 23:22:58.108162 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:58.108144 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123\": container with ID starting with 84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123 not found: ID does not exist" containerID="84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123" Nov 02 23:22:58.108228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.108198 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123"} err="failed to get container status \"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123\": rpc error: code = NotFound desc = could not find container \"84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123\": container with ID starting with 84508bef3a040c8842feb20ec156883d75efea79f8cceef5c34cf494ee251123 not found: ID does not exist" Nov 02 23:22:58.257529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.255210 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:22:58.260751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.260726 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee5b4-openshift-default-64d48d448c-l5zr5"] Nov 02 23:22:58.653297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.653266 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:22:58.653528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.653462 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" podUID="275aba89-b42e-44b6-abe6-a32c3db6ff13" containerName="maas-api" containerID="cri-o://b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a" gracePeriod=30 Nov 02 23:22:58.753957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.753922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:58.962191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.950587 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:58.967029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:58.966726 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:22:59.064127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064098 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\") pod \"275aba89-b42e-44b6-abe6-a32c3db6ff13\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " Nov 02 23:22:59.064321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064139 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\") pod \"275aba89-b42e-44b6-abe6-a32c3db6ff13\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " Nov 02 23:22:59.064321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064200 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\" (UniqueName: \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\") pod \"275aba89-b42e-44b6-abe6-a32c3db6ff13\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " Nov 02 23:22:59.064431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064415 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") pod \"275aba89-b42e-44b6-abe6-a32c3db6ff13\" (UID: \"275aba89-b42e-44b6-abe6-a32c3db6ff13\") " Nov 02 23:22:59.064596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064556 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca" pod "275aba89-b42e-44b6-abe6-a32c3db6ff13" (UID: "275aba89-b42e-44b6-abe6-a32c3db6ff13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:59.064910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.064817 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275aba89-b42e-44b6-abe6-a32c3db6ff13-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.066450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.066425 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs" pod "275aba89-b42e-44b6-abe6-a32c3db6ff13" (UID: "275aba89-b42e-44b6-abe6-a32c3db6ff13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:59.066571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.066554 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2" pod "275aba89-b42e-44b6-abe6-a32c3db6ff13" (UID: "275aba89-b42e-44b6-abe6-a32c3db6ff13"). InnerVolumeSpecName "kube-api-access-dnsn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:59.066621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.066591 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls" pod "275aba89-b42e-44b6-abe6-a32c3db6ff13" (UID: "275aba89-b42e-44b6-abe6-a32c3db6ff13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:22:59.104887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.104849 4494 generic.go:356] "Generic (PLEG): container finished" podID="275aba89-b42e-44b6-abe6-a32c3db6ff13" containerID="b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a" exitCode=0 Nov 02 23:22:59.105366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.104955 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" Nov 02 23:22:59.105366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.104983 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" event={"ID":"275aba89-b42e-44b6-abe6-a32c3db6ff13","Type":"ContainerDied","Data":"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a"} Nov 02 23:22:59.105366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.105045 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g" event={"ID":"275aba89-b42e-44b6-abe6-a32c3db6ff13","Type":"ContainerDied","Data":"0907ca0bb900e6a852f2456596af830046ffcb2c31d83c74dfc687db3f21f7cd"} Nov 02 23:22:59.105366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.105074 4494 scope.go:122] "RemoveContainer" containerID="b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a" Nov 02 23:22:59.114401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.113375 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:59.120407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.119149 4494 scope.go:122] "RemoveContainer" containerID="b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a" Nov 02 23:22:59.120407 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:22:59.119624 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a\": container with ID starting with b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a not found: ID does not exist" containerID="b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a" Nov 02 23:22:59.120407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.119659 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a"} err="failed to get container status \"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a\": rpc error: code = NotFound desc = could not find container \"b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a\": container with ID starting with b9daa2eb158f61cd41cd9af11059190abd8f1c15f61c8f1f18c1417aa1f9cd1a not found: ID does not exist" Nov 02 23:22:59.166036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.166001 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.166036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.166028 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275aba89-b42e-44b6-abe6-a32c3db6ff13-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.166036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.166040 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsn2\" (UniqueName: \"kubernetes.io/projected/275aba89-b42e-44b6-abe6-a32c3db6ff13-kube-api-access-dnsn2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.169217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.169191 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:22:59.169432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.169410 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" podUID="c341b213-a714-438f-8c3f-94eb28234934" containerName="payload-processing" containerID="cri-o://ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a" gracePeriod=30 Nov 02 23:22:59.221055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.220968 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:22:59.221252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.221201 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" podUID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" containerName="payload-pre-processing" containerID="cri-o://51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe" gracePeriod=30 Nov 02 23:22:59.271033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.270995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:59.280575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.280534 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:22:59.284973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.284943 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b1c72-657488949d-4ld6g"] Nov 02 23:22:59.423747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.423712 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:22:59.489911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.489883 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:22:59.503036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.503014 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:22:59.570495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.570465 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\") pod \"c341b213-a714-438f-8c3f-94eb28234934\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " Nov 02 23:22:59.570696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.570530 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\" (UniqueName: \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\") pod \"c341b213-a714-438f-8c3f-94eb28234934\" (UID: \"c341b213-a714-438f-8c3f-94eb28234934\") " Nov 02 23:22:59.570919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.570887 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume" pod "c341b213-a714-438f-8c3f-94eb28234934" (UID: "c341b213-a714-438f-8c3f-94eb28234934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:59.572814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.572793 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb" pod "c341b213-a714-438f-8c3f-94eb28234934" (UID: "c341b213-a714-438f-8c3f-94eb28234934"). InnerVolumeSpecName "kube-api-access-4l6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:59.671822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.671502 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\" (UniqueName: \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\") pod \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " Nov 02 23:22:59.671822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.671631 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\") pod \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\" (UID: \"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3\") " Nov 02 23:22:59.672247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.672080 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume" pod "2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" (UID: "2c7fa98e-44fa-4baf-bc1a-3d886bd790e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:59.672247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.672113 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c341b213-a714-438f-8c3f-94eb28234934-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.672247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.672134 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l6mb\" (UniqueName: \"kubernetes.io/projected/c341b213-a714-438f-8c3f-94eb28234934-kube-api-access-4l6mb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.674529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.674503 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq" pod "2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" (UID: "2c7fa98e-44fa-4baf-bc1a-3d886bd790e3"). InnerVolumeSpecName "kube-api-access-9dgmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:59.742186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.741718 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275aba89-b42e-44b6-abe6-a32c3db6ff13" path="/var/lib/kubelet/pods/275aba89-b42e-44b6-abe6-a32c3db6ff13/volumes" Nov 02 23:22:59.753197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.748600 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" path="/var/lib/kubelet/pods/f4202e77-afe6-4be4-91ae-7de75abeaadb/volumes" Nov 02 23:22:59.784917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.777391 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:22:59.784917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:22:59.777425 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgmq\" (UniqueName: \"kubernetes.io/projected/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3-kube-api-access-9dgmq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:00.119587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.119481 4494 generic.go:356] "Generic (PLEG): container finished" podID="c341b213-a714-438f-8c3f-94eb28234934" containerID="ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a" exitCode=0 Nov 02 23:23:00.119587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.119548 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" Nov 02 23:23:00.119587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.119567 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" event={"ID":"c341b213-a714-438f-8c3f-94eb28234934","Type":"ContainerDied","Data":"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a"} Nov 02 23:23:00.120166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.119638 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd" event={"ID":"c341b213-a714-438f-8c3f-94eb28234934","Type":"ContainerDied","Data":"cb0d16ca13bc0722f0d445e3256a8683905cd743b4a68522095a1fd39436560e"} Nov 02 23:23:00.120166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.119681 4494 scope.go:122] "RemoveContainer" containerID="ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a" Nov 02 23:23:00.121722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.121695 4494 generic.go:356] "Generic (PLEG): container finished" podID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" containerID="51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe" exitCode=0 Nov 02 23:23:00.121867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.121842 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" event={"ID":"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3","Type":"ContainerDied","Data":"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe"} Nov 02 23:23:00.121945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.121760 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" Nov 02 23:23:00.122104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.121901 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2" event={"ID":"2c7fa98e-44fa-4baf-bc1a-3d886bd790e3","Type":"ContainerDied","Data":"24456842b91e360ef78a4b9f01e440102affb715bec0bbae1db0d1738f9606ee"} Nov 02 23:23:00.130638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.130621 4494 scope.go:122] "RemoveContainer" containerID="ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a" Nov 02 23:23:00.130929 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:00.130911 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a\": container with ID starting with ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a not found: ID does not exist" containerID="ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a" Nov 02 23:23:00.130986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.130940 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a"} err="failed to get container status \"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a\": rpc error: code = NotFound desc = could not find container \"ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a\": container with ID starting with ed0214ae4ab2ddf6721c16bb366720ec199313968b733cd34ee580b140865e3a not found: ID does not exist" Nov 02 23:23:00.130986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.130959 4494 scope.go:122] "RemoveContainer" containerID="51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe" Nov 02 23:23:00.140081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.140063 4494 scope.go:122] "RemoveContainer" containerID="51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe" Nov 02 23:23:00.140359 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:00.140341 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe\": container with ID starting with 51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe not found: ID does not exist" containerID="51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe" Nov 02 23:23:00.140433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.140369 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe"} err="failed to get container status \"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe\": rpc error: code = NotFound desc = could not find container \"51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe\": container with ID starting with 51e0dc118da453d16aba20ea789a467f95a9f1b8cce0ebf1a2390b35ea2f56fe not found: ID does not exist" Nov 02 23:23:00.288216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.288167 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:23:00.291238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.291216 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b1c72-5bbc97f54b-kkznd"] Nov 02 23:23:00.408313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.408215 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:23:00.413110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.413078 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b1c72-5cd6f688cf-5v4j2"] Nov 02 23:23:00.768523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:00.768489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:01.166811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:01.166779 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:01.333868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:01.333829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:01.695212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:01.695158 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" path="/var/lib/kubelet/pods/2c7fa98e-44fa-4baf-bc1a-3d886bd790e3/volumes" Nov 02 23:23:01.695685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:01.695664 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c341b213-a714-438f-8c3f-94eb28234934" path="/var/lib/kubelet/pods/c341b213-a714-438f-8c3f-94eb28234934/volumes" Nov 02 23:23:02.802297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:02.802258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:02.987876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:02.987841 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:03.216560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.216531 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:23:03.217097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217084 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c341b213-a714-438f-8c3f-94eb28234934" containerName="payload-processing" Nov 02 23:23:03.217140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217098 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c341b213-a714-438f-8c3f-94eb28234934" containerName="payload-processing" Nov 02 23:23:03.217140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217117 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.217140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217122 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.217263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217144 4494 cpu_manager.go:414] "containerMap: removing container" podUID="275aba89-b42e-44b6-abe6-a32c3db6ff13" containerName="maas-api" Nov 02 23:23:03.217263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217150 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275aba89-b42e-44b6-abe6-a32c3db6ff13" containerName="maas-api" Nov 02 23:23:03.217263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217164 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" containerName="payload-pre-processing" Nov 02 23:23:03.217263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217182 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" containerName="payload-pre-processing" Nov 02 23:23:03.217263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217264 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c341b213-a714-438f-8c3f-94eb28234934" containerName="payload-processing" Nov 02 23:23:03.217415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217274 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.217415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217285 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="275aba89-b42e-44b6-abe6-a32c3db6ff13" containerName="maas-api" Nov 02 23:23:03.217415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.217292 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7fa98e-44fa-4baf-bc1a-3d886bd790e3" containerName="payload-pre-processing" Nov 02 23:23:03.221937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.221913 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.278072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.278040 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:23:03.347313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.338164 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.367662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.367625 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttcv\" (UniqueName: \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.368079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.366430 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-03fe6e\"" Nov 02 23:23:03.379345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.379312 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-03fe6e-dockercfg-qp654\"" Nov 02 23:23:03.388781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.388702 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:23:03.389884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.389857 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.390118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.390103 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.390434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.390420 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4202e77-afe6-4be4-91ae-7de75abeaadb" containerName="istio-proxy" Nov 02 23:23:03.422222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.421766 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.434099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.434067 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:23:03.468809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.468726 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.468809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.468797 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.469040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.468913 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttcv\" (UniqueName: \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.469040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.468975 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhsk\" (UniqueName: \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.469474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.469456 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.530856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.530826 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttcv\" (UniqueName: \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\") pod \"payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.535758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.535730 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:03.570255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.570212 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhsk\" (UniqueName: \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.570407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.570386 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.571077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.571053 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.618341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.618310 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:03.632642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.632614 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhsk\" (UniqueName: \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\") pod \"payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.738752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.738670 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:03.926767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:03.926734 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:23:03.927280 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:03.927250 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b54e5a_571d_48f6_bc06_63c4fba211f0.slice/crio-367a1f0ed514f4535e8c92d70ac3c789c3865e44797ac1b7e77065464a94d1bc WatchSource:0}: Error finding container 367a1f0ed514f4535e8c92d70ac3c789c3865e44797ac1b7e77065464a94d1bc: Status 404 returned error can't find the container with id 367a1f0ed514f4535e8c92d70ac3c789c3865e44797ac1b7e77065464a94d1bc Nov 02 23:23:04.104581 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.104541 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:23:04.104809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.104782 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" containerID="cri-o://6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13" gracePeriod=30 Nov 02 23:23:04.116570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.116513 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:23:04.121881 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:04.121823 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c23b809_0a23_4e88_8c66_7ff20ca80376.slice/crio-37287da17557aa47e3d076bf7a5063380545310533d9f62928b67978fdda8dd4 WatchSource:0}: Error finding container 37287da17557aa47e3d076bf7a5063380545310533d9f62928b67978fdda8dd4: Status 404 returned error can't find the container with id 37287da17557aa47e3d076bf7a5063380545310533d9f62928b67978fdda8dd4 Nov 02 23:23:04.150037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.150010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:04.156962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.156935 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" event={"ID":"a3b54e5a-571d-48f6-bc06-63c4fba211f0","Type":"ContainerStarted","Data":"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6"} Nov 02 23:23:04.157088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.156984 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" event={"ID":"a3b54e5a-571d-48f6-bc06-63c4fba211f0","Type":"ContainerStarted","Data":"367a1f0ed514f4535e8c92d70ac3c789c3865e44797ac1b7e77065464a94d1bc"} Nov 02 23:23:04.157088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.157016 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:04.158359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.158337 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" event={"ID":"0c23b809-0a23-4e88-8c66-7ff20ca80376","Type":"ContainerStarted","Data":"37287da17557aa47e3d076bf7a5063380545310533d9f62928b67978fdda8dd4"} Nov 02 23:23:04.433723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:04.433678 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" podStartSLOduration=1.433661821 podStartE2EDuration="1.433661821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:04.433280763 +0000 UTC m=+2098.863150627" watchObservedRunningTime="2026-11-02 23:23:04.433661821 +0000 UTC m=+2098.863531682" Nov 02 23:23:05.166188 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.166141 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" event={"ID":"0c23b809-0a23-4e88-8c66-7ff20ca80376","Type":"ContainerStarted","Data":"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf"} Nov 02 23:23:05.166631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.166367 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:05.283497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.283432 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" podStartSLOduration=2.283410506 podStartE2EDuration="2.283410506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:05.283221544 +0000 UTC m=+2099.713091406" watchObservedRunningTime="2026-11-02 23:23:05.283410506 +0000 UTC m=+2099.713280369" Nov 02 23:23:05.604556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.604466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:05.993327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.993294 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:23:05.993503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:05.993347 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:23:06.088048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.088017 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:23:06.092401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.092378 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.155644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.155602 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-11f7a8-openshift-default-dockercfg-67zld\"" Nov 02 23:23:06.185344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.185312 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:06.200285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.200253 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.254237 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.254330 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.254370 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxb8\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.254407 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.254540 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.266283 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.266375 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.266437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.267294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.266504 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.367979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.367945 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368000 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368071 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368112 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368142 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxb8\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368194 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368303 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368338 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368386 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368605 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368608 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.368760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368689 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.369094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368848 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.369094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.368874 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.370587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.370566 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.371118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.371090 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.429936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.429890 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.430158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.430139 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxb8\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\") pod \"e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.443902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.443866 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:06.708654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.707641 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:06.733371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:06.732920 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:07.156355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.156331 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:23:07.157101 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:07.157081 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f188c60_f265_4f8b_82da_726d3f2cd3b9.slice/crio-34ce314f3dda1b0f4c24ec87a0570551f2bb9f8797dd0676ea89aeed6947a5d0 WatchSource:0}: Error finding container 34ce314f3dda1b0f4c24ec87a0570551f2bb9f8797dd0676ea89aeed6947a5d0: Status 404 returned error can't find the container with id 34ce314f3dda1b0f4c24ec87a0570551f2bb9f8797dd0676ea89aeed6947a5d0 Nov 02 23:23:07.159750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.159711 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:07.159865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.159789 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:07.159865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.159832 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:07.182766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.182741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerStarted","Data":"34ce314f3dda1b0f4c24ec87a0570551f2bb9f8797dd0676ea89aeed6947a5d0"} Nov 02 23:23:07.270418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:07.270383 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:08.190373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:08.190339 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerStarted","Data":"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29"} Nov 02 23:23:08.321975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:08.321946 4494 scope.go:122] "RemoveContainer" containerID="e50e3cac944025909dc1d60867e10412f165d42a70c9f69dfc9182404dad46b5" Nov 02 23:23:08.378065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:08.378009 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podStartSLOduration=2.3779915320000002 podStartE2EDuration="2.377991532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:08.363138131 +0000 UTC m=+2102.793008005" watchObservedRunningTime="2026-11-02 23:23:08.377991532 +0000 UTC m=+2102.807861393" Nov 02 23:23:08.709020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:08.708982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:08.714601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:08.714566 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:09.200113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.200077 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:09.201240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.201222 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:23:09.392125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.392097 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:23:09.506494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506457 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506507 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506562 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506647 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506739 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506790 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506890 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.506935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506918 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:09.507059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506935 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.507059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.506960 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:09.507059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507035 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\") pod \"2d35d456-ed5a-45f2-b436-93d6d7d95369\" (UID: \"2d35d456-ed5a-45f2-b436-93d6d7d95369\") " Nov 02 23:23:09.507265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507114 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:09.507536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507503 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:09.507804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507778 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:09.507913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507827 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.507913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507842 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.507913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507851 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.507913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.507860 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d35d456-ed5a-45f2-b436-93d6d7d95369-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.509222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.509201 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "kube-api-access-gczlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:09.509317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.509223 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:09.509408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.509383 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:09.509460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.509400 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token" pod "2d35d456-ed5a-45f2-b436-93d6d7d95369" (UID: "2d35d456-ed5a-45f2-b436-93d6d7d95369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:09.609727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.609688 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.609727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.609726 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.609939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.609743 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.609939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.609756 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d35d456-ed5a-45f2-b436-93d6d7d95369-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:09.609939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:09.609771 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczlq\" (UniqueName: \"kubernetes.io/projected/2d35d456-ed5a-45f2-b436-93d6d7d95369-kube-api-access-gczlq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:23:10.082095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.081995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:10.150903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.146105 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:23:10.207792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.207763 4494 generic.go:356] "Generic (PLEG): container finished" podID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerID="6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13" exitCode=0 Nov 02 23:23:10.207977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.207826 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" Nov 02 23:23:10.207977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.207853 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerDied","Data":"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13"} Nov 02 23:23:10.207977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.207909 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7" event={"ID":"2d35d456-ed5a-45f2-b436-93d6d7d95369","Type":"ContainerDied","Data":"468c08b326ea5f0aa85432ab28d71f53812e07a9df8a0d85cda8d6a30812110b"} Nov 02 23:23:10.207977 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.207924 4494 scope.go:122] "RemoveContainer" containerID="6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13" Nov 02 23:23:10.217672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.217655 4494 scope.go:122] "RemoveContainer" containerID="612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405" Nov 02 23:23:10.226963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.226934 4494 scope.go:122] "RemoveContainer" containerID="6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13" Nov 02 23:23:10.227260 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:10.227242 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13\": container with ID starting with 6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13 not found: ID does not exist" containerID="6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13" Nov 02 23:23:10.227327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.227272 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13"} err="failed to get container status \"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13\": rpc error: code = NotFound desc = could not find container \"6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13\": container with ID starting with 6119dc977247e4780e4329022c117dee34cde4236403fe4a9f89343740778d13 not found: ID does not exist" Nov 02 23:23:10.227327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.227290 4494 scope.go:122] "RemoveContainer" containerID="612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405" Nov 02 23:23:10.227541 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:10.227525 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405\": container with ID starting with 612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405 not found: ID does not exist" containerID="612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405" Nov 02 23:23:10.227589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.227547 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405"} err="failed to get container status \"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405\": rpc error: code = NotFound desc = could not find container \"612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405\": container with ID starting with 612caffdd2b0802bfe90fc73cff5a29b76557b280f759ed42d64a37b1d610405 not found: ID does not exist" Nov 02 23:23:10.378785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.378699 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:23:10.429108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.429079 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2b1c72-openshift-default-59f478b847-2fks7"] Nov 02 23:23:10.606836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:10.606795 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:11.148532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.148499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:11.324016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.323986 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:11.468731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.468702 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:11.663779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.663741 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:23:11.664370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.664355 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:11.664414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.664371 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:11.664515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.664507 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:11.669039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.669021 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.694108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.694077 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" path="/var/lib/kubelet/pods/2d35d456-ed5a-45f2-b436-93d6d7d95369/volumes" Nov 02 23:23:11.725640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.725574 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:23:11.752548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.752519 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.753153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.752899 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhf7\" (UniqueName: \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.771804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.771776 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1cb1f8-dockercfg-6dp2d\"" Nov 02 23:23:11.771936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.771812 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1cb1f8\"" Nov 02 23:23:11.854421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.854393 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.854599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.854518 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhf7\" (UniqueName: \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.855056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.855033 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.916239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.916210 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhf7\" (UniqueName: \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\") pod \"payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:11.983473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:11.983382 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:12.362162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.362138 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:23:12.363466 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:12.363440 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfbb708_e07d_4dbb_98b8_455683b4d368.slice/crio-b78f7b2d0625087d159e1a0b0149dbd5e528c7a28261938b3a4b86cd5ff71788 WatchSource:0}: Error finding container b78f7b2d0625087d159e1a0b0149dbd5e528c7a28261938b3a4b86cd5ff71788: Status 404 returned error can't find the container with id b78f7b2d0625087d159e1a0b0149dbd5e528c7a28261938b3a4b86cd5ff71788 Nov 02 23:23:12.415686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.415661 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:23:12.416487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.416467 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:12.416487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.416487 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:12.416606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.416595 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d35d456-ed5a-45f2-b436-93d6d7d95369" containerName="istio-proxy" Nov 02 23:23:12.421226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.421209 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.461784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.461756 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btf6j\" (UniqueName: \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.462015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.461996 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.477419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.477389 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:23:12.564833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.564260 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.564833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.564383 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btf6j\" (UniqueName: \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.572604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.565444 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.627661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.627590 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btf6j\" (UniqueName: \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\") pod \"payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:12.744032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:12.743992 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:13.121951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.121908 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:23:13.122131 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:13.122110 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a8408f_d4a1_4d72_ba21_03f33baccd01.slice/crio-cea674dec7ac2c534764162cb24c830c59d66818059f3e28a7662ad0b221e94f WatchSource:0}: Error finding container cea674dec7ac2c534764162cb24c830c59d66818059f3e28a7662ad0b221e94f: Status 404 returned error can't find the container with id cea674dec7ac2c534764162cb24c830c59d66818059f3e28a7662ad0b221e94f Nov 02 23:23:13.229355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.229318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" event={"ID":"33a8408f-d4a1-4d72-ba21-03f33baccd01","Type":"ContainerStarted","Data":"56e907e639d52cdf2aee6a7ceb8609e4531baf53d588500d11f2eb3b59002baa"} Nov 02 23:23:13.229526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.229378 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:13.229526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.229405 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" event={"ID":"33a8408f-d4a1-4d72-ba21-03f33baccd01","Type":"ContainerStarted","Data":"cea674dec7ac2c534764162cb24c830c59d66818059f3e28a7662ad0b221e94f"} Nov 02 23:23:13.231247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.231216 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" event={"ID":"fcfbb708-e07d-4dbb-98b8-455683b4d368","Type":"ContainerStarted","Data":"ca6c97c375bfd8a07e92a88279a539e0f62d4cf234d5bd0998484e48fd04c0a8"} Nov 02 23:23:13.231358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.231270 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" event={"ID":"fcfbb708-e07d-4dbb-98b8-455683b4d368","Type":"ContainerStarted","Data":"b78f7b2d0625087d159e1a0b0149dbd5e528c7a28261938b3a4b86cd5ff71788"} Nov 02 23:23:13.231426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.231386 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:13.350067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.350014 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" podStartSLOduration=1.3500007219999999 podStartE2EDuration="1.350000722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:13.347768325 +0000 UTC m=+2107.777638187" watchObservedRunningTime="2026-11-02 23:23:13.350000722 +0000 UTC m=+2107.779870611" Nov 02 23:23:13.497641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.497571 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" podStartSLOduration=2.497546286 podStartE2EDuration="2.497546286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:13.48156025 +0000 UTC m=+2107.911430112" watchObservedRunningTime="2026-11-02 23:23:13.497546286 +0000 UTC m=+2107.927416148" Nov 02 23:23:13.665065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:13.665030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:14.382263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.382232 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:14.529153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.529118 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:14.741955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.741916 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:23:14.746096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.746069 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.799474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.799441 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-11f7a8\"" Nov 02 23:23:14.813959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.813931 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289ft\" (UniqueName: \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.814145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.814081 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.853339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.853268 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-11f7a8-dockercfg-jnb8p\"" Nov 02 23:23:14.859022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.858971 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:23:14.947383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.947345 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:23:14.966213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.951021 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.966733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.966703 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289ft\" (UniqueName: \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.967446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.960259 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:14.971394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.971369 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:14.995837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:14.995762 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:23:15.040002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.039967 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-289ft\" (UniqueName: \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\") pod \"payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:15.062660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.062632 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:15.068477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.068455 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpxd\" (UniqueName: \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.068613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.068595 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.169043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.169012 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:23:15.169407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.169380 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpxd\" (UniqueName: \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.169533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.169510 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.170450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.170424 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.231662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.231636 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpxd\" (UniqueName: \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\") pod \"payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.286708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.286619 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:15.439965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.439939 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:23:15.440939 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:15.440906 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80139230_5c41_4b6e_9e82_f9f49d67bda0.slice/crio-87dc156a9876397f42530f616fa3ea55004a139dc5944ed8e60eb2d80cbdde01 WatchSource:0}: Error finding container 87dc156a9876397f42530f616fa3ea55004a139dc5944ed8e60eb2d80cbdde01: Status 404 returned error can't find the container with id 87dc156a9876397f42530f616fa3ea55004a139dc5944ed8e60eb2d80cbdde01 Nov 02 23:23:15.676163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:15.676141 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:23:15.677443 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:15.677421 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554fe342_5956_41cb_bf7d_00fd24ee6ffd.slice/crio-124557986b607ce26f775051ef8eda30a0c767bf90d799c6640887b18b9a67f1 WatchSource:0}: Error finding container 124557986b607ce26f775051ef8eda30a0c767bf90d799c6640887b18b9a67f1: Status 404 returned error can't find the container with id 124557986b607ce26f775051ef8eda30a0c767bf90d799c6640887b18b9a67f1 Nov 02 23:23:16.177828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.177801 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:23:16.255213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.255159 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" event={"ID":"80139230-5c41-4b6e-9e82-f9f49d67bda0","Type":"ContainerStarted","Data":"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30"} Nov 02 23:23:16.255412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.255238 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" event={"ID":"80139230-5c41-4b6e-9e82-f9f49d67bda0","Type":"ContainerStarted","Data":"87dc156a9876397f42530f616fa3ea55004a139dc5944ed8e60eb2d80cbdde01"} Nov 02 23:23:16.255412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.255269 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:16.257091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.257067 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" event={"ID":"554fe342-5956-41cb-bf7d-00fd24ee6ffd","Type":"ContainerStarted","Data":"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7"} Nov 02 23:23:16.257232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.257107 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" event={"ID":"554fe342-5956-41cb-bf7d-00fd24ee6ffd","Type":"ContainerStarted","Data":"124557986b607ce26f775051ef8eda30a0c767bf90d799c6640887b18b9a67f1"} Nov 02 23:23:16.257232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.257132 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:16.412989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.412934 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" podStartSLOduration=2.412918943 podStartE2EDuration="2.412918943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:16.411759843 +0000 UTC m=+2110.841629703" watchObservedRunningTime="2026-11-02 23:23:16.412918943 +0000 UTC m=+2110.842788814" Nov 02 23:23:16.550251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:16.545558 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" podStartSLOduration=2.545531572 podStartE2EDuration="2.545531572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:16.533001827 +0000 UTC m=+2110.962871690" watchObservedRunningTime="2026-11-02 23:23:16.545531572 +0000 UTC m=+2110.975401436" Nov 02 23:23:17.564858 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:17.564774 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:23:19.112002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:19.111958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:22.810297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:22.810255 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:22.954037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:22.954003 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:24.143487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:24.143453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:24.241628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:24.241598 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:23:24.241796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:24.241759 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:23:27.266268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:27.266239 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:23:27.266742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:27.266315 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:23:27.541353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:27.541265 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:28.040844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.040816 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:23:28.046441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.046422 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.102046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.102005 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-014ab4-openshift-default-dockercfg-wqdvr\"" Nov 02 23:23:28.133163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133124 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133216 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133320 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7zv\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133371 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133444 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133525 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133552 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133588 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133653 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.133841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.133817 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:28.154583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.154552 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:23:28.246199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.234983 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.246550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.246520 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.246725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.246703 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7zv\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.256231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.246885 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.256502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.256477 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.261015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.260986 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.272869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.272826 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.272914 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.272965 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.273472 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.254566 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.273737 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.273774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.257321 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.274040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.237777 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.284082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.284042 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.285918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.285887 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.322686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.322581 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.323770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.323718 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7zv\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\") pod \"e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.343551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.343379 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:28.360989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.360951 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:28.813475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.813445 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:23:28.813978 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:28.813953 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f66ece_b910_4cab_8940_19d95bb7e4e2.slice/crio-683e95466d31f5a4bc7b99a7ac1acb7bb70e99898568d04948c69b4c5506a348 WatchSource:0}: Error finding container 683e95466d31f5a4bc7b99a7ac1acb7bb70e99898568d04948c69b4c5506a348: Status 404 returned error can't find the container with id 683e95466d31f5a4bc7b99a7ac1acb7bb70e99898568d04948c69b4c5506a348 Nov 02 23:23:28.816720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.816684 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:28.816794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.816749 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:28.816794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:28.816778 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:23:29.133974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.133871 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:29.345254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.345213 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerStarted","Data":"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86"} Nov 02 23:23:29.345762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.345280 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerStarted","Data":"683e95466d31f5a4bc7b99a7ac1acb7bb70e99898568d04948c69b4c5506a348"} Nov 02 23:23:29.361978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.361955 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:29.367316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.367294 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:29.468850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:29.468794 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podStartSLOduration=2.46878033 podStartE2EDuration="2.46878033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:29.466578687 +0000 UTC m=+2123.896448548" watchObservedRunningTime="2026-11-02 23:23:29.46878033 +0000 UTC m=+2123.898650192" Nov 02 23:23:30.355877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:30.355838 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:30.357086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:30.357062 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:23:32.272909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:32.272873 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:32.443129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:32.443091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:32.651276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:32.651187 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:34.423908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:34.423873 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:35.503328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:35.503285 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:35.678639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:35.678606 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:41.442327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:41.442286 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:41.593688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:41.593657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:41.740359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:41.740278 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:46.524623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:46.524576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:47.560215 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:23:47.560153 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:23:51.651061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:51.651023 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:55.178539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:55.178507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:55.670370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:55.670336 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:56.769780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.769748 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:23:56.775208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.775160 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:56.832068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.832039 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:23:56.878103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.878075 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-92072f\"" Nov 02 23:23:56.878103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.878076 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-92072f-dockercfg-p56x9\"" Nov 02 23:23:56.922352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.922322 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcvh\" (UniqueName: \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:56.922517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:56.922368 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.023284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.023142 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcvh\" (UniqueName: \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.023284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.023259 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.024113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.024082 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.084952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.084925 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcvh\" (UniqueName: \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\") pod \"payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.089273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.089246 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:57.469299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.469267 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:23:57.470620 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:57.470594 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43fec33_7562_41bf_be7f_047c85ca0a4a.slice/crio-4a4f4932388e7fc0c847508e39d080ab69889b3d7f24e0a9cd8f3aa3d256c3cb WatchSource:0}: Error finding container 4a4f4932388e7fc0c847508e39d080ab69889b3d7f24e0a9cd8f3aa3d256c3cb: Status 404 returned error can't find the container with id 4a4f4932388e7fc0c847508e39d080ab69889b3d7f24e0a9cd8f3aa3d256c3cb Nov 02 23:23:57.529161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.529130 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" event={"ID":"b43fec33-7562-41bf-be7f-047c85ca0a4a","Type":"ContainerStarted","Data":"4a4f4932388e7fc0c847508e39d080ab69889b3d7f24e0a9cd8f3aa3d256c3cb"} Nov 02 23:23:57.560192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.560143 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:23:57.564527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.564506 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.622228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.622197 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:23:57.630979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.630953 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.631147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.631030 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcbg\" (UniqueName: \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.732116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.732028 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcbg\" (UniqueName: \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.732345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.732228 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.732889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.732870 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.794041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.793999 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcbg\" (UniqueName: \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\") pod \"payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:57.880619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:57.880582 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:58.261930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.261884 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:23:58.262783 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:23:58.262753 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4831fc7c_0618_4b2c_a3f3_b801b04f7b4f.slice/crio-05cac84f4be24bbbee739184ead0216de5fe8a26c96f644f2ef7ed43d9f20b28 WatchSource:0}: Error finding container 05cac84f4be24bbbee739184ead0216de5fe8a26c96f644f2ef7ed43d9f20b28: Status 404 returned error can't find the container with id 05cac84f4be24bbbee739184ead0216de5fe8a26c96f644f2ef7ed43d9f20b28 Nov 02 23:23:58.431872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.431831 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:23:58.540368 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.540277 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" event={"ID":"b43fec33-7562-41bf-be7f-047c85ca0a4a","Type":"ContainerStarted","Data":"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3"} Nov 02 23:23:58.540556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.540378 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:23:58.543250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.543216 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" event={"ID":"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f","Type":"ContainerStarted","Data":"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c"} Nov 02 23:23:58.543400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.543275 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" event={"ID":"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f","Type":"ContainerStarted","Data":"05cac84f4be24bbbee739184ead0216de5fe8a26c96f644f2ef7ed43d9f20b28"} Nov 02 23:23:58.543925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.543903 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:23:58.660993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.660931 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" podStartSLOduration=2.6609125970000003 podStartE2EDuration="2.660912597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:58.657944509 +0000 UTC m=+2153.087814392" watchObservedRunningTime="2026-11-02 23:23:58.660912597 +0000 UTC m=+2153.090782461" Nov 02 23:23:58.788117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:23:58.780964 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" podStartSLOduration=1.78094173 podStartE2EDuration="1.78094173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:58.779596348 +0000 UTC m=+2153.209466211" watchObservedRunningTime="2026-11-02 23:23:58.78094173 +0000 UTC m=+2153.210811593" Nov 02 23:24:01.389101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:01.389065 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:24:03.258838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:03.258798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:03.407694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:03.407664 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:03.555398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:03.555286 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:07.617559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:07.617486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:08.463671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:08.463590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:09.553143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:09.553109 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:24:10.558231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:10.558199 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:24:11.672243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:11.670947 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": read tcp 10.133.0.2:59534->10.133.0.174:15021: read: connection reset by peer" start-of-body= Nov 02 23:24:11.672243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:11.671030 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": read tcp 10.133.0.2:59534->10.133.0.174:15021: read: connection reset by peer" Nov 02 23:24:12.657130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.657093 4494 generic.go:356] "Generic (PLEG): container finished" podID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerID="5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b" exitCode=0 Nov 02 23:24:12.657345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.657166 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerDied","Data":"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b"} Nov 02 23:24:12.657626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.657612 4494 scope.go:122] "RemoveContainer" containerID="5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b" Nov 02 23:24:12.659192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.659155 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:12.659256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.659236 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:12.659293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:12.659269 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:13.104416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:13.104377 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:13.505099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:13.505061 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:13.666606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:13.666576 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerStarted","Data":"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1"} Nov 02 23:24:14.301743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:14.301710 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:24:14.302276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:14.302078 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:24:14.302276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:14.302132 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:24:15.302395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:15.302357 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:24:15.302810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:15.302423 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:24:16.302743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:16.302684 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:24:16.303192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:16.302766 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:24:17.302148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:17.302110 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:24:17.302365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:17.302202 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:24:17.589656 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:17.589565 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:24:18.302494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:18.302453 4494 patch_prober.go:29] interesting pod/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:24:18.302686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:18.302521 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 02 23:24:18.976419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:18.976383 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:19.305899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.305822 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:24:19.306232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.306212 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:24:19.306756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.306737 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:24:19.900768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.900735 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:24:19.905158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.905137 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:19.959657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.959626 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-11f7a8-serving-cert\"" Nov 02 23:24:19.964007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:19.963981 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:24:20.028137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.028099 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.028635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.028299 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvkb\" (UniqueName: \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.028635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.028347 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.028635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.028487 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.096162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.096128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:20.129637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.129589 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvkb\" (UniqueName: \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.129796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.129659 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.129796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.129703 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.129796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.129786 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.129945 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:20.129914 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-serving-cert: secret "maas-api-e2e-praxis-a-11f7a8-serving-cert" not found Nov 02 23:24:20.129999 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:20.129988 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls podName:7be01df7-2de9-460b-a122-d6796ed35412 nodeName:}" failed. No retries permitted until 2026-11-02 23:24:20.629968417 +0000 UTC m=+2175.059838259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls") pod "maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" (UID: "7be01df7-2de9-460b-a122-d6796ed35412") : secret "maas-api-e2e-praxis-a-11f7a8-serving-cert" not found Nov 02 23:24:20.130337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.130319 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.132365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.132349 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.194242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.194205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvkb\" (UniqueName: \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.282874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.282841 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:20.428973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.428940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:20.645975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.645868 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.648586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.648563 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") pod \"maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:20.818900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:20.818874 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:21.196383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:21.196340 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:24:21.197396 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:24:21.197370 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be01df7_2de9_460b_a122_d6796ed35412.slice/crio-bb05aba55f6ccfce8b44c6ea924501360a7b59043a1ba9a7b0ce6ab9a3b6c437 WatchSource:0}: Error finding container bb05aba55f6ccfce8b44c6ea924501360a7b59043a1ba9a7b0ce6ab9a3b6c437: Status 404 returned error can't find the container with id bb05aba55f6ccfce8b44c6ea924501360a7b59043a1ba9a7b0ce6ab9a3b6c437 Nov 02 23:24:21.198920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:21.198903 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:24:21.718154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:21.718117 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" event={"ID":"7be01df7-2de9-460b-a122-d6796ed35412","Type":"ContainerStarted","Data":"bb05aba55f6ccfce8b44c6ea924501360a7b59043a1ba9a7b0ce6ab9a3b6c437"} Nov 02 23:24:22.727767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:22.727729 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" event={"ID":"7be01df7-2de9-460b-a122-d6796ed35412","Type":"ContainerStarted","Data":"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8"} Nov 02 23:24:22.728283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:22.727816 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:22.863643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:22.863589 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" podStartSLOduration=3.113214769 podStartE2EDuration="3.863573071s" totalImagesPullingTime="750.358302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:24:22.861791827 +0000 UTC m=+2177.291661696" watchObservedRunningTime="2026-11-02 23:24:22.863573071 +0000 UTC m=+2177.293442937" Nov 02 23:24:25.580279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:25.580240 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:28.766388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:28.766357 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:24:30.562856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:30.562826 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:31.820053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.819974 4494 generic.go:356] "Generic (PLEG): container finished" podID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerID="b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486" exitCode=0 Nov 02 23:24:31.820535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.820046 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerDied","Data":"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486"} Nov 02 23:24:31.820535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.820445 4494 scope.go:122] "RemoveContainer" containerID="b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486" Nov 02 23:24:31.822082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.822054 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:31.822154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.822125 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:31.822208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:31.822159 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:24:32.830014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:32.829977 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerStarted","Data":"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508"} Nov 02 23:24:33.582401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:33.582370 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:24:33.582594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:33.582486 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:24:33.582594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:33.582530 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 23:24:34.582765 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:34.582734 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:24:34.583208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:34.582791 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 23:24:35.484909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:35.484878 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:35.589138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:35.588598 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:24:35.589138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:35.588671 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 23:24:36.582987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:36.582947 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:24:36.583180 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:36.583013 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 23:24:37.583043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:37.583014 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 02 23:24:37.583521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:37.583077 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 02 23:24:38.587680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:38.587646 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:24:38.588156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:38.588138 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:24:38.588827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:38.588799 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:24:40.429216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:40.429151 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:45.187111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:45.187081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:45.343144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:45.343109 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:45.544276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:45.544150 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:47.561505 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:47.561428 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:24:47.848854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:47.848767 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:24:49.617321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:49.617287 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:24:49.617759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:49.617526 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" podUID="1d18244f-8a13-4338-b991-35e041a0790b" containerName="maas-api" containerID="cri-o://c84c288a87f9be05c23d10273fe74984e0096ab2f953e6f8ec98a5feb22bda34" gracePeriod=30 Nov 02 23:24:49.718136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:49.718071 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:49.971696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:49.971666 4494 generic.go:356] "Generic (PLEG): container finished" podID="1d18244f-8a13-4338-b991-35e041a0790b" containerID="c84c288a87f9be05c23d10273fe74984e0096ab2f953e6f8ec98a5feb22bda34" exitCode=0 Nov 02 23:24:49.971862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:49.971740 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" event={"ID":"1d18244f-8a13-4338-b991-35e041a0790b","Type":"ContainerDied","Data":"c84c288a87f9be05c23d10273fe74984e0096ab2f953e6f8ec98a5feb22bda34"} Nov 02 23:24:50.008423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.008393 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:24:50.108020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.107987 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:24:50.108258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.108233 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" podUID="a0e0952d-c068-44e2-ab73-c5ea24cad284" containerName="payload-processing" containerID="cri-o://efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b" gracePeriod=30 Nov 02 23:24:50.130683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.129957 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\") pod \"1d18244f-8a13-4338-b991-35e041a0790b\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " Nov 02 23:24:50.130976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.130954 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\" (UniqueName: \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\") pod \"1d18244f-8a13-4338-b991-35e041a0790b\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " Nov 02 23:24:50.131611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.131584 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\") pod \"1d18244f-8a13-4338-b991-35e041a0790b\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " Nov 02 23:24:50.136629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.136477 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca" pod "1d18244f-8a13-4338-b991-35e041a0790b" (UID: "1d18244f-8a13-4338-b991-35e041a0790b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:50.136920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.136893 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\") pod \"1d18244f-8a13-4338-b991-35e041a0790b\" (UID: \"1d18244f-8a13-4338-b991-35e041a0790b\") " Nov 02 23:24:50.152197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.140356 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd" pod "1d18244f-8a13-4338-b991-35e041a0790b" (UID: "1d18244f-8a13-4338-b991-35e041a0790b"). InnerVolumeSpecName "kube-api-access-tlxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:50.152197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.141012 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxpd\" (UniqueName: \"kubernetes.io/projected/1d18244f-8a13-4338-b991-35e041a0790b-kube-api-access-tlxpd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.152197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.141039 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d18244f-8a13-4338-b991-35e041a0790b-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.166779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.166656 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs" pod "1d18244f-8a13-4338-b991-35e041a0790b" (UID: "1d18244f-8a13-4338-b991-35e041a0790b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:24:50.174708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.174666 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls" pod "1d18244f-8a13-4338-b991-35e041a0790b" (UID: "1d18244f-8a13-4338-b991-35e041a0790b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:24:50.177970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.177942 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:24:50.178270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.178149 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" podUID="3391d6d7-a137-41cf-9144-181a86cb647c" containerName="payload-pre-processing" containerID="cri-o://4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1" gracePeriod=30 Nov 02 23:24:50.270843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.270364 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.270843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.270408 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d18244f-8a13-4338-b991-35e041a0790b-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.282582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.280912 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:50.424607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.424144 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:50.511710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.511683 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:24:50.575916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.575890 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:24:50.577634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.577606 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\") pod \"3391d6d7-a137-41cf-9144-181a86cb647c\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " Nov 02 23:24:50.577954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.577923 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\" (UniqueName: \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\") pod \"3391d6d7-a137-41cf-9144-181a86cb647c\" (UID: \"3391d6d7-a137-41cf-9144-181a86cb647c\") " Nov 02 23:24:50.578403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.578360 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume" pod "3391d6d7-a137-41cf-9144-181a86cb647c" (UID: "3391d6d7-a137-41cf-9144-181a86cb647c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:50.578552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.578531 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3391d6d7-a137-41cf-9144-181a86cb647c-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.581350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.581322 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d" pod "3391d6d7-a137-41cf-9144-181a86cb647c" (UID: "3391d6d7-a137-41cf-9144-181a86cb647c"). InnerVolumeSpecName "kube-api-access-zvm8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:50.684867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.683795 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\" (UniqueName: \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\") pod \"a0e0952d-c068-44e2-ab73-c5ea24cad284\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " Nov 02 23:24:50.684867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.684029 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\") pod \"a0e0952d-c068-44e2-ab73-c5ea24cad284\" (UID: \"a0e0952d-c068-44e2-ab73-c5ea24cad284\") " Nov 02 23:24:50.684867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.684557 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvm8d\" (UniqueName: \"kubernetes.io/projected/3391d6d7-a137-41cf-9144-181a86cb647c-kube-api-access-zvm8d\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.696196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.685806 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume" pod "a0e0952d-c068-44e2-ab73-c5ea24cad284" (UID: "a0e0952d-c068-44e2-ab73-c5ea24cad284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:50.783495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.783448 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd" pod "a0e0952d-c068-44e2-ab73-c5ea24cad284" (UID: "a0e0952d-c068-44e2-ab73-c5ea24cad284"). InnerVolumeSpecName "kube-api-access-t2drd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:50.791220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.785527 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2drd\" (UniqueName: \"kubernetes.io/projected/a0e0952d-c068-44e2-ab73-c5ea24cad284-kube-api-access-t2drd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.791220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.785555 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e0952d-c068-44e2-ab73-c5ea24cad284-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:50.981152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.981125 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" Nov 02 23:24:50.981369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.981120 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg" event={"ID":"1d18244f-8a13-4338-b991-35e041a0790b","Type":"ContainerDied","Data":"de53ae506f4dd164dc105130d55e97c7f0617de1cc980bde882fb13cba06d593"} Nov 02 23:24:50.981369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.981260 4494 scope.go:122] "RemoveContainer" containerID="c84c288a87f9be05c23d10273fe74984e0096ab2f953e6f8ec98a5feb22bda34" Nov 02 23:24:50.984466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.984441 4494 generic.go:356] "Generic (PLEG): container finished" podID="a0e0952d-c068-44e2-ab73-c5ea24cad284" containerID="efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b" exitCode=0 Nov 02 23:24:50.984571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.984488 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" Nov 02 23:24:50.984571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.984530 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" event={"ID":"a0e0952d-c068-44e2-ab73-c5ea24cad284","Type":"ContainerDied","Data":"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b"} Nov 02 23:24:50.984664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.984588 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd" event={"ID":"a0e0952d-c068-44e2-ab73-c5ea24cad284","Type":"ContainerDied","Data":"e7adde87892560ede90f6b19cd24a3d9e99edbd8098ef1abac0dd67e3fe2caba"} Nov 02 23:24:50.986165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.986142 4494 generic.go:356] "Generic (PLEG): container finished" podID="3391d6d7-a137-41cf-9144-181a86cb647c" containerID="4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1" exitCode=0 Nov 02 23:24:50.986309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.986209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" event={"ID":"3391d6d7-a137-41cf-9144-181a86cb647c","Type":"ContainerDied","Data":"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1"} Nov 02 23:24:50.986309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.986224 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" Nov 02 23:24:50.986309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.986255 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58" event={"ID":"3391d6d7-a137-41cf-9144-181a86cb647c","Type":"ContainerDied","Data":"d0505deef3885f12e9e82d2160d8b1391e55805bf99cce74225851f11319f7b1"} Nov 02 23:24:50.996761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:50.996741 4494 scope.go:122] "RemoveContainer" containerID="efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b" Nov 02 23:24:51.006603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.006585 4494 scope.go:122] "RemoveContainer" containerID="efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b" Nov 02 23:24:51.006893 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:51.006875 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b\": container with ID starting with efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b not found: ID does not exist" containerID="efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b" Nov 02 23:24:51.006937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.006903 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b"} err="failed to get container status \"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b\": rpc error: code = NotFound desc = could not find container \"efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b\": container with ID starting with efa746a28d45fdecd4b2318ee4bbeb2ff5b9c7dafe6c49fbccdc064c7a23602b not found: ID does not exist" Nov 02 23:24:51.006937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.006921 4494 scope.go:122] "RemoveContainer" containerID="4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1" Nov 02 23:24:51.015900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.015883 4494 scope.go:122] "RemoveContainer" containerID="4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1" Nov 02 23:24:51.016162 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:51.016142 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1\": container with ID starting with 4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1 not found: ID does not exist" containerID="4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1" Nov 02 23:24:51.016234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.016185 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1"} err="failed to get container status \"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1\": rpc error: code = NotFound desc = could not find container \"4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1\": container with ID starting with 4626a98de074cd2acd5dcc21ed61a6da2054bc1c0c0bbe98ba352547d6e085e1 not found: ID does not exist" Nov 02 23:24:51.163707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.163675 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:24:51.167385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.167364 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d00785-9975c4b66-rrwxd"] Nov 02 23:24:51.278251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.278122 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:24:51.281406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.281375 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d00785-746bcc85c8-qqlkg"] Nov 02 23:24:51.382375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.382345 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:51.411800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.411769 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:24:51.414428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.414409 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d00785-856b9c6cff-zrz58"] Nov 02 23:24:51.528793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.528702 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:51.671504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.671461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:51.708004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.707966 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d18244f-8a13-4338-b991-35e041a0790b" path="/var/lib/kubelet/pods/1d18244f-8a13-4338-b991-35e041a0790b/volumes" Nov 02 23:24:51.708500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.708465 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3391d6d7-a137-41cf-9144-181a86cb647c" path="/var/lib/kubelet/pods/3391d6d7-a137-41cf-9144-181a86cb647c/volumes" Nov 02 23:24:51.708843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:51.708828 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e0952d-c068-44e2-ab73-c5ea24cad284" path="/var/lib/kubelet/pods/a0e0952d-c068-44e2-ab73-c5ea24cad284/volumes" Nov 02 23:24:52.987298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:52.987261 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:53.471344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:53.471315 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:24:53.471584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:53.471560 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" containerID="cri-o://8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f" gracePeriod=30 Nov 02 23:24:53.479729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:53.479700 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:53.804713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:53.804631 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:53.804893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:53.804702 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:54.258091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:54.258043 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:56.445767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:56.445724 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:56.595595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:56.595547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:57.148261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:57.148154 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:24:58.752073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.752014 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:24:58.826467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826431 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.826675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826495 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.826675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826637 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.826808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826717 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.826867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826794 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:58.826933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826891 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:58.826933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826891 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.827041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.826983 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.827041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827013 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.827151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827123 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.827399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827211 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\") pod \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\" (UID: \"e4137d1d-8c99-463a-a5df-cdb4880a2e61\") " Nov 02 23:24:58.827399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827371 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:58.827506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827413 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:58.827569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.827545 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:58.829224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.828386 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.829224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.828427 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.829224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.828442 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.829224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.828456 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.829224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.828469 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.831696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.830831 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:58.831696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.830856 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "kube-api-access-2s46s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:58.841857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.841737 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:58.841857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.841781 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token" pod "e4137d1d-8c99-463a-a5df-cdb4880a2e61" (UID: "e4137d1d-8c99-463a-a5df-cdb4880a2e61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:58.929991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.929953 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.929991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.929994 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s46s\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-kube-api-access-2s46s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.930251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.930012 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:58.930251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:58.930026 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4137d1d-8c99-463a-a5df-cdb4880a2e61-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:24:59.048760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.048667 4494 generic.go:356] "Generic (PLEG): container finished" podID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerID="8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f" exitCode=0 Nov 02 23:24:59.048760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.048735 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerDied","Data":"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f"} Nov 02 23:24:59.048760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.048750 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" Nov 02 23:24:59.049881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.048832 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn" event={"ID":"e4137d1d-8c99-463a-a5df-cdb4880a2e61","Type":"ContainerDied","Data":"b672da4e93211ca673136fe7cf541e8a6b79c97a71b61dbe709c3fb63558a26d"} Nov 02 23:24:59.049881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.048863 4494 scope.go:122] "RemoveContainer" containerID="8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f" Nov 02 23:24:59.061406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.061386 4494 scope.go:122] "RemoveContainer" containerID="94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18" Nov 02 23:24:59.071261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.071235 4494 scope.go:122] "RemoveContainer" containerID="8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f" Nov 02 23:24:59.071544 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:59.071522 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f\": container with ID starting with 8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f not found: ID does not exist" containerID="8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f" Nov 02 23:24:59.071604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.071555 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f"} err="failed to get container status \"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f\": rpc error: code = NotFound desc = could not find container \"8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f\": container with ID starting with 8e1161841b5d1392b8060dde32bdf314b48662774c9b6bedab42638a872ea06f not found: ID does not exist" Nov 02 23:24:59.071604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.071577 4494 scope.go:122] "RemoveContainer" containerID="94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18" Nov 02 23:24:59.071831 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:24:59.071811 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18\": container with ID starting with 94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18 not found: ID does not exist" containerID="94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18" Nov 02 23:24:59.071894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.071842 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18"} err="failed to get container status \"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18\": rpc error: code = NotFound desc = could not find container \"94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18\": container with ID starting with 94b24b5733936c19413fa7d947cb52507ea6ce744be179f10150171be5930d18 not found: ID does not exist" Nov 02 23:24:59.216759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.216669 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:24:59.221060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.221033 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d00785-openshift-default-78c5d6c695-vkzfn"] Nov 02 23:24:59.694431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.694394 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" path="/var/lib/kubelet/pods/e4137d1d-8c99-463a-a5df-cdb4880a2e61/volumes" Nov 02 23:24:59.830647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:24:59.830568 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:00.057960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.057916 4494 generic.go:356] "Generic (PLEG): container finished" podID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerID="402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29" exitCode=0 Nov 02 23:25:00.058142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.057988 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerDied","Data":"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29"} Nov 02 23:25:00.058437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.058422 4494 scope.go:122] "RemoveContainer" containerID="402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29" Nov 02 23:25:00.059946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.059907 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:00.060071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.059982 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:00.060071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.060014 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:00.341064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:00.341028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:01.065744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.065710 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerStarted","Data":"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009"} Nov 02 23:25:01.133855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.133817 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:01.708693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.708665 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:25:01.708895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.708854 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:01.708965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.708918 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:01.761952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:01.761914 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:02.127735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.127630 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:25:02.128228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.127870 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" podUID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" containerName="payload-processing" containerID="cri-o://6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c" gracePeriod=30 Nov 02 23:25:02.184339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.184309 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:25:02.184541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.184522 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" podUID="b43fec33-7562-41bf-be7f-047c85ca0a4a" containerName="payload-pre-processing" containerID="cri-o://c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3" gracePeriod=30 Nov 02 23:25:02.281403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.281366 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:02.429140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.429040 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:02.491963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.491940 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:25:02.500645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.500616 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\" (UniqueName: \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\") pod \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " Nov 02 23:25:02.500797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.500683 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\") pod \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\" (UID: \"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f\") " Nov 02 23:25:02.501299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.501264 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume" pod "4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" (UID: "4831fc7c-0618-4b2c-a3f3-b801b04f7b4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:02.503063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.503040 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg" pod "4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" (UID: "4831fc7c-0618-4b2c-a3f3-b801b04f7b4f"). InnerVolumeSpecName "kube-api-access-gpcbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:02.524953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.524930 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:25:02.601689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.601655 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\" (UniqueName: \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\") pod \"b43fec33-7562-41bf-be7f-047c85ca0a4a\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " Nov 02 23:25:02.601903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.601803 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\") pod \"b43fec33-7562-41bf-be7f-047c85ca0a4a\" (UID: \"b43fec33-7562-41bf-be7f-047c85ca0a4a\") " Nov 02 23:25:02.602125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.602107 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcbg\" (UniqueName: \"kubernetes.io/projected/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-kube-api-access-gpcbg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.602219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.602126 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.602219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.602158 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume" pod "b43fec33-7562-41bf-be7f-047c85ca0a4a" (UID: "b43fec33-7562-41bf-be7f-047c85ca0a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:02.603940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.603920 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh" pod "b43fec33-7562-41bf-be7f-047c85ca0a4a" (UID: "b43fec33-7562-41bf-be7f-047c85ca0a4a"). InnerVolumeSpecName "kube-api-access-9tcvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:02.712979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.712937 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43fec33-7562-41bf-be7f-047c85ca0a4a-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:02.722270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.713283 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:02.722270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.714670 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:02.724214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:02.724162 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tcvh\" (UniqueName: \"kubernetes.io/projected/b43fec33-7562-41bf-be7f-047c85ca0a4a-kube-api-access-9tcvh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:03.121620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.121522 4494 generic.go:356] "Generic (PLEG): container finished" podID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" containerID="6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c" exitCode=0 Nov 02 23:25:03.121620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.121584 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" Nov 02 23:25:03.121620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.121609 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" event={"ID":"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f","Type":"ContainerDied","Data":"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c"} Nov 02 23:25:03.121915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.121661 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs" event={"ID":"4831fc7c-0618-4b2c-a3f3-b801b04f7b4f","Type":"ContainerDied","Data":"05cac84f4be24bbbee739184ead0216de5fe8a26c96f644f2ef7ed43d9f20b28"} Nov 02 23:25:03.121915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.121685 4494 scope.go:122] "RemoveContainer" containerID="6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c" Nov 02 23:25:03.123087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.123068 4494 generic.go:356] "Generic (PLEG): container finished" podID="b43fec33-7562-41bf-be7f-047c85ca0a4a" containerID="c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3" exitCode=0 Nov 02 23:25:03.123219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.123123 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" Nov 02 23:25:03.123219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.123138 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" event={"ID":"b43fec33-7562-41bf-be7f-047c85ca0a4a","Type":"ContainerDied","Data":"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3"} Nov 02 23:25:03.123219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.123186 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l" event={"ID":"b43fec33-7562-41bf-be7f-047c85ca0a4a","Type":"ContainerDied","Data":"4a4f4932388e7fc0c847508e39d080ab69889b3d7f24e0a9cd8f3aa3d256c3cb"} Nov 02 23:25:03.158643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.158499 4494 scope.go:122] "RemoveContainer" containerID="6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c" Nov 02 23:25:03.158946 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:03.158895 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c\": container with ID starting with 6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c not found: ID does not exist" containerID="6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c" Nov 02 23:25:03.159015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.158934 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c"} err="failed to get container status \"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c\": rpc error: code = NotFound desc = could not find container \"6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c\": container with ID starting with 6618e99d8e33013d4b493b58d8ea8c1d4b0d6f657dd0be393c1dc8b06768346c not found: ID does not exist" Nov 02 23:25:03.159015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.158960 4494 scope.go:122] "RemoveContainer" containerID="c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3" Nov 02 23:25:03.168578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.168557 4494 scope.go:122] "RemoveContainer" containerID="c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3" Nov 02 23:25:03.168838 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:03.168819 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3\": container with ID starting with c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3 not found: ID does not exist" containerID="c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3" Nov 02 23:25:03.168892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.168846 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3"} err="failed to get container status \"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3\": rpc error: code = NotFound desc = could not find container \"c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3\": container with ID starting with c142b751156afde49758b989f440da97e5e04ba91e77e8f1453fde0aff50bab3 not found: ID does not exist" Nov 02 23:25:03.289540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.289504 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:25:03.293255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.293230 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92072f-58487d8f64-pcdfs"] Nov 02 23:25:03.409408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.409371 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:25:03.413140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.413111 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-92072f-7bbc5c78b7-j9z2l"] Nov 02 23:25:03.595917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.595883 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:03.696470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.696385 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" path="/var/lib/kubelet/pods/4831fc7c-0618-4b2c-a3f3-b801b04f7b4f/volumes" Nov 02 23:25:03.696813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.696798 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43fec33-7562-41bf-be7f-047c85ca0a4a" path="/var/lib/kubelet/pods/b43fec33-7562-41bf-be7f-047c85ca0a4a/volumes" Nov 02 23:25:03.708451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.708417 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:03.708636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:03.708474 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:04.708478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:04.708443 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:04.708918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:04.708506 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:05.708675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:05.708644 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:05.709150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:05.708696 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:06.188715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:06.188684 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:06.341767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:06.341697 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:06.708589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:06.708552 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 02 23:25:06.708773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:06.708607 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 02 23:25:07.357726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.357648 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:07.713555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.713526 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:25:07.714068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.713878 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:25:07.714638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.714620 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:25:07.826606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.826543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:07.972374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:07.972289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:08.292888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.292794 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:25:08.293098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.293067 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" containerID="cri-o://fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6" gracePeriod=30 Nov 02 23:25:08.324541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.324505 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:08.691906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.691871 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:25:08.798947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.798601 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:25:08.889976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:08.889944 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:09.063343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:09.063164 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:09.214989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:09.214956 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:09.805581 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:09.805552 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:10.352954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.352913 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:25:10.356112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356087 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1d18244f-8a13-4338-b991-35e041a0790b" containerName="maas-api" Nov 02 23:25:10.356284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356212 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d18244f-8a13-4338-b991-35e041a0790b" containerName="maas-api" Nov 02 23:25:10.356365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356294 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356304 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356333 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a0e0952d-c068-44e2-ab73-c5ea24cad284" containerName="payload-processing" Nov 02 23:25:10.356365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356342 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e0952d-c068-44e2-ab73-c5ea24cad284" containerName="payload-processing" Nov 02 23:25:10.356365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356361 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356369 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356404 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3391d6d7-a137-41cf-9144-181a86cb647c" containerName="payload-pre-processing" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356415 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3391d6d7-a137-41cf-9144-181a86cb647c" containerName="payload-pre-processing" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356429 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b43fec33-7562-41bf-be7f-047c85ca0a4a" containerName="payload-pre-processing" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356437 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43fec33-7562-41bf-be7f-047c85ca0a4a" containerName="payload-pre-processing" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356459 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" containerName="payload-processing" Nov 02 23:25:10.356634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356481 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" containerName="payload-processing" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356657 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d18244f-8a13-4338-b991-35e041a0790b" containerName="maas-api" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356681 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356689 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e0952d-c068-44e2-ab73-c5ea24cad284" containerName="payload-processing" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356716 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4137d1d-8c99-463a-a5df-cdb4880a2e61" containerName="istio-proxy" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356728 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3391d6d7-a137-41cf-9144-181a86cb647c" containerName="payload-pre-processing" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356739 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43fec33-7562-41bf-be7f-047c85ca0a4a" containerName="payload-pre-processing" Nov 02 23:25:10.356999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.356749 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4831fc7c-0618-4b2c-a3f3-b801b04f7b4f" containerName="payload-processing" Nov 02 23:25:10.363130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.362446 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.414098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.414071 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:25:10.444938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.444811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:10.460004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.459972 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d530c9-openshift-default-dockercfg-c8bm5\"" Nov 02 23:25:10.499608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.499573 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.499608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.499607 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmjq\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.499876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.499636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.499876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.499847 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.500014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.499956 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.500149 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.500091 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.500258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.500165 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.500258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.500228 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.500375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.500258 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.597142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.596320 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:10.602465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.602429 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.602691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.602668 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmjq\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.602836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.602821 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.603012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.602963 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.603148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.603133 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.603353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.603335 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.619451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.619408 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.619689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.619671 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.619858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.619843 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.620535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.620511 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.620943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.620921 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.621648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.621627 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.622287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.622266 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.624617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.624252 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.649866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.649570 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.649866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.649804 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.695207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.694255 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.698568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.698534 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmjq\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\") pod \"e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:10.977531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:10.977498 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:11.429219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:11.429193 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:25:11.429810 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:25:11.429783 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a9f3e4_08d0_499f_a5a9_b691be3792f3.slice/crio-ca6843fa6323e765c79d45aa30e326c54ed9196e937d8e1a05c974d1fe9d6e20 WatchSource:0}: Error finding container ca6843fa6323e765c79d45aa30e326c54ed9196e937d8e1a05c974d1fe9d6e20: Status 404 returned error can't find the container with id ca6843fa6323e765c79d45aa30e326c54ed9196e937d8e1a05c974d1fe9d6e20 Nov 02 23:25:11.453329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:11.451648 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:11.453329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:11.451733 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:11.453329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:11.451779 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:12.190584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.190548 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:25:12.191079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.190608 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:25:12.213273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.213242 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerStarted","Data":"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca"} Nov 02 23:25:12.213441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.213306 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerStarted","Data":"ca6843fa6323e765c79d45aa30e326c54ed9196e937d8e1a05c974d1fe9d6e20"} Nov 02 23:25:12.379719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.379654 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podStartSLOduration=2.379636052 podStartE2EDuration="2.379636052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:12.375759667 +0000 UTC m=+2226.805629528" watchObservedRunningTime="2026-11-02 23:25:12.379636052 +0000 UTC m=+2226.809505913" Nov 02 23:25:12.978301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.978266 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:12.983824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:12.983800 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:13.108842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.108764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:13.221258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.221227 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:13.222294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.222268 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:25:13.669670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.669605 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:25:13.692859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.692826 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" podUID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" containerName="payload-processing" containerID="cri-o://134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b" gracePeriod=25 Nov 02 23:25:13.697805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.697773 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.697950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.697849 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.697996 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698035 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698091 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698146 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698213 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698255 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698326 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\") pod \"61f8fad4-e3ae-492c-9719-cd76c0825085\" (UID: \"61f8fad4-e3ae-492c-9719-cd76c0825085\") " Nov 02 23:25:13.698745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698423 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:13.699472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698784 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.699472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.698828 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:13.699472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.699251 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:13.699472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.699297 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:13.699472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.699435 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:13.701389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.701361 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:25:13.701389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.701379 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "kube-api-access-rpbjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:13.702425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.702400 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:25:13.703045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.703020 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token" pod "61f8fad4-e3ae-492c-9719-cd76c0825085" (UID: "61f8fad4-e3ae-492c-9719-cd76c0825085"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:13.799797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799764 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.799797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799798 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799813 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbjx\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-kube-api-access-rpbjx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799828 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799840 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799853 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799865 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f8fad4-e3ae-492c-9719-cd76c0825085-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799879 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f8fad4-e3ae-492c-9719-cd76c0825085-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:13.800065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.799919 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" podUID="62009735-e9ab-40c2-aec8-6fe861465d58" containerName="payload-pre-processing" containerID="cri-o://0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5" gracePeriod=25 Nov 02 23:25:13.967891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:13.967861 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:25:14.002950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.002741 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\") pod \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " Nov 02 23:25:14.002950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.002855 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\" (UniqueName: \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\") pod \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\" (UID: \"f96a5f15-0ed9-42d3-8295-a27bb2980a78\") " Nov 02 23:25:14.003873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.003841 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume" pod "f96a5f15-0ed9-42d3-8295-a27bb2980a78" (UID: "f96a5f15-0ed9-42d3-8295-a27bb2980a78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:14.007737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.007639 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6" pod "f96a5f15-0ed9-42d3-8295-a27bb2980a78" (UID: "f96a5f15-0ed9-42d3-8295-a27bb2980a78"). InnerVolumeSpecName "kube-api-access-5r9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:14.091565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.091534 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:25:14.104646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.104615 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\" (UniqueName: \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\") pod \"62009735-e9ab-40c2-aec8-6fe861465d58\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " Nov 02 23:25:14.104824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.104739 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\") pod \"62009735-e9ab-40c2-aec8-6fe861465d58\" (UID: \"62009735-e9ab-40c2-aec8-6fe861465d58\") " Nov 02 23:25:14.105106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.105079 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume" pod "62009735-e9ab-40c2-aec8-6fe861465d58" (UID: "62009735-e9ab-40c2-aec8-6fe861465d58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:14.105106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.105098 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a5f15-0ed9-42d3-8295-a27bb2980a78-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:14.105236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.105114 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r9k6\" (UniqueName: \"kubernetes.io/projected/f96a5f15-0ed9-42d3-8295-a27bb2980a78-kube-api-access-5r9k6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:14.107242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.107213 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l" pod "62009735-e9ab-40c2-aec8-6fe861465d58" (UID: "62009735-e9ab-40c2-aec8-6fe861465d58"). InnerVolumeSpecName "kube-api-access-p2k4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:14.206055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.205979 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009735-e9ab-40c2-aec8-6fe861465d58-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:14.206055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.206005 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2k4l\" (UniqueName: \"kubernetes.io/projected/62009735-e9ab-40c2-aec8-6fe861465d58-kube-api-access-p2k4l\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:14.230093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.230050 4494 generic.go:356] "Generic (PLEG): container finished" podID="62009735-e9ab-40c2-aec8-6fe861465d58" containerID="0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5" exitCode=0 Nov 02 23:25:14.230637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.230098 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" Nov 02 23:25:14.230637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.230124 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" event={"ID":"62009735-e9ab-40c2-aec8-6fe861465d58","Type":"ContainerDied","Data":"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5"} Nov 02 23:25:14.230637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.230221 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk" event={"ID":"62009735-e9ab-40c2-aec8-6fe861465d58","Type":"ContainerDied","Data":"380b5c15ad00dc4f7d98b74f292f82d2481b35de7b2c2e2d75b9ecbd06e00a57"} Nov 02 23:25:14.230637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.230256 4494 scope.go:122] "RemoveContainer" containerID="0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5" Nov 02 23:25:14.233160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.233111 4494 generic.go:356] "Generic (PLEG): container finished" podID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerID="fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6" exitCode=0 Nov 02 23:25:14.233300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.233211 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" Nov 02 23:25:14.233300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.233288 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerDied","Data":"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6"} Nov 02 23:25:14.233431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.233333 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c" event={"ID":"61f8fad4-e3ae-492c-9719-cd76c0825085","Type":"ContainerDied","Data":"670ad36bbee5b53498e240d47763c1b5cef84ff3aa1cc9e82a05563e48950b2b"} Nov 02 23:25:14.234912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.234885 4494 generic.go:356] "Generic (PLEG): container finished" podID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" containerID="134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b" exitCode=0 Nov 02 23:25:14.235853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.235681 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" Nov 02 23:25:14.240631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.240599 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" event={"ID":"f96a5f15-0ed9-42d3-8295-a27bb2980a78","Type":"ContainerDied","Data":"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b"} Nov 02 23:25:14.241026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.240863 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj" event={"ID":"f96a5f15-0ed9-42d3-8295-a27bb2980a78","Type":"ContainerDied","Data":"58102166d0c560a8d42432f90e8d703f590903177cbb112d1694d335a44a04e2"} Nov 02 23:25:14.250147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.250125 4494 scope.go:122] "RemoveContainer" containerID="0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5" Nov 02 23:25:14.251291 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:14.251270 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5\": container with ID starting with 0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5 not found: ID does not exist" containerID="0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5" Nov 02 23:25:14.251403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.251297 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5"} err="failed to get container status \"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5\": rpc error: code = NotFound desc = could not find container \"0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5\": container with ID starting with 0a00ad6a8108018d0d246933b0d7b4942e6aa8778225669d82e61054ff8b9dd5 not found: ID does not exist" Nov 02 23:25:14.251403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.251314 4494 scope.go:122] "RemoveContainer" containerID="fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6" Nov 02 23:25:14.263620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.263598 4494 scope.go:122] "RemoveContainer" containerID="06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005" Nov 02 23:25:14.275579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.275554 4494 scope.go:122] "RemoveContainer" containerID="fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6" Nov 02 23:25:14.275893 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:14.275873 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6\": container with ID starting with fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6 not found: ID does not exist" containerID="fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6" Nov 02 23:25:14.275992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.275910 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6"} err="failed to get container status \"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6\": rpc error: code = NotFound desc = could not find container \"fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6\": container with ID starting with fd69c7cb227614242b1484e44e7b3df4015d6ebacf191edc7472a351f828afb6 not found: ID does not exist" Nov 02 23:25:14.275992 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.275929 4494 scope.go:122] "RemoveContainer" containerID="06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005" Nov 02 23:25:14.276263 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:14.276242 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005\": container with ID starting with 06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005 not found: ID does not exist" containerID="06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005" Nov 02 23:25:14.276342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.276272 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005"} err="failed to get container status \"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005\": rpc error: code = NotFound desc = could not find container \"06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005\": container with ID starting with 06a1e0db0da41b30ac9ae9f0e6329bd338bd3ea8a81540570816d9826d363005 not found: ID does not exist" Nov 02 23:25:14.276342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.276297 4494 scope.go:122] "RemoveContainer" containerID="134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b" Nov 02 23:25:14.286738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.286716 4494 scope.go:122] "RemoveContainer" containerID="134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b" Nov 02 23:25:14.286999 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:14.286977 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b\": container with ID starting with 134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b not found: ID does not exist" containerID="134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b" Nov 02 23:25:14.287068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.287011 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b"} err="failed to get container status \"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b\": rpc error: code = NotFound desc = could not find container \"134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b\": container with ID starting with 134227487ee178a09945bb0ec9276ba81b1086e9530e7a97123f82e3cde7dd4b not found: ID does not exist" Nov 02 23:25:14.359446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.359414 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:14.398298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.398264 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:25:14.408940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.408901 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb3c46-7694d4c4c5-9bzmk"] Nov 02 23:25:14.512826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.512746 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:25:14.516403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.516375 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-92072f-openshift-default-5b854c667c-8nn8c"] Nov 02 23:25:14.630101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.630064 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:25:14.633004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:14.632977 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb3c46-587686dd97-mjxhj"] Nov 02 23:25:15.696258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:15.696227 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" path="/var/lib/kubelet/pods/61f8fad4-e3ae-492c-9719-cd76c0825085/volumes" Nov 02 23:25:15.696652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:15.696636 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62009735-e9ab-40c2-aec8-6fe861465d58" path="/var/lib/kubelet/pods/62009735-e9ab-40c2-aec8-6fe861465d58/volumes" Nov 02 23:25:15.696999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:15.696987 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" path="/var/lib/kubelet/pods/f96a5f15-0ed9-42d3-8295-a27bb2980a78/volumes" Nov 02 23:25:15.867104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:15.867066 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:16.091078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:16.090994 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:16.292731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:16.292703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:16.398330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:16.398294 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:25:16.399325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:16.398349 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:25:16.929576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:16.929549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:17.266659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.266571 4494 generic.go:356] "Generic (PLEG): container finished" podID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerID="6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86" exitCode=0 Nov 02 23:25:17.266659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.266641 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerDied","Data":"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86"} Nov 02 23:25:17.267041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.267025 4494 scope.go:122] "RemoveContainer" containerID="6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86" Nov 02 23:25:17.268447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.268420 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:17.268553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.268482 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:17.268553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:17.268513 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:17.576085 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:17.575992 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:25:18.058385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.058351 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:18.275105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.275071 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerStarted","Data":"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749"} Nov 02 23:25:18.361710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.361620 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:25:18.361872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.361839 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:18.361947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.361918 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:18.584401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.584371 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:25:18.584984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.584968 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:18.585055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.584986 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:18.585055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585010 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" containerName="payload-processing" Nov 02 23:25:18.585055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585016 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" containerName="payload-processing" Nov 02 23:25:18.585055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585030 4494 cpu_manager.go:414] "containerMap: removing container" podUID="62009735-e9ab-40c2-aec8-6fe861465d58" containerName="payload-pre-processing" Nov 02 23:25:18.585055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585036 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62009735-e9ab-40c2-aec8-6fe861465d58" containerName="payload-pre-processing" Nov 02 23:25:18.585333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585141 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:18.585333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585150 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:18.585333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585160 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96a5f15-0ed9-42d3-8295-a27bb2980a78" containerName="payload-processing" Nov 02 23:25:18.585333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.585184 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="62009735-e9ab-40c2-aec8-6fe861465d58" containerName="payload-pre-processing" Nov 02 23:25:18.587516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.587494 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.644378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.644272 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a2f549-openshift-default-dockercfg-g2vnt\"" Nov 02 23:25:18.670196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.662232 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699197 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699271 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699300 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5tc\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699338 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699373 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699439 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699499 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699524 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.718222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.699587 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.720935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.720786 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870086 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870237 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870304 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5tc\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870360 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870397 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870486 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870517 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.870538 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.871301 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.871553 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.871758 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.872428 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.894252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.875481 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.895070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.895004 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.898027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.898005 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.959293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.944655 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:18.971132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:18.971064 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5tc\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\") pod \"e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:19.207126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.206438 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:19.224182 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.224135 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:19.363619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.363585 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:19.363788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.363652 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:19.668150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.668126 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:25:19.671992 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:25:19.671965 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f88f573_34a1_4832_857c_eff3178bbeb6.slice/crio-1a136effcfcbb009f15c071137d136b992a3271686d3ccbabcfa64a7c156aef3 WatchSource:0}: Error finding container 1a136effcfcbb009f15c071137d136b992a3271686d3ccbabcfa64a7c156aef3: Status 404 returned error can't find the container with id 1a136effcfcbb009f15c071137d136b992a3271686d3ccbabcfa64a7c156aef3 Nov 02 23:25:19.674595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.674565 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:19.674679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.674627 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:19.674679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.674658 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:25:19.685596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.685572 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:19.883974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:19.883883 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:20.039405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.037995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:20.092864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.092825 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:25:20.093588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.093564 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:20.093588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.093588 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f8fad4-e3ae-492c-9719-cd76c0825085" containerName="istio-proxy" Nov 02 23:25:20.096001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.095981 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.101872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.101850 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.102039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.102023 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js86v\" (UniqueName: \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.164969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.158964 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:25:20.164969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.164522 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d530c9\"" Nov 02 23:25:20.164969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.164685 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d530c9-dockercfg-mdkwt\"" Nov 02 23:25:20.211205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.206548 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js86v\" (UniqueName: \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.211205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.207056 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.211205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.207937 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.298043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.297953 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:25:20.343782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.326881 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js86v\" (UniqueName: \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\") pod \"payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.374121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.374086 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:25:20.374956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.374930 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.376525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.376445 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:20.376772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.376710 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:20.389199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.387554 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerStarted","Data":"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9"} Nov 02 23:25:20.389199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.387636 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerStarted","Data":"1a136effcfcbb009f15c071137d136b992a3271686d3ccbabcfa64a7c156aef3"} Nov 02 23:25:20.409636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.409608 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:20.412694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.412673 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.412799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.412786 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbnr\" (UniqueName: \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.513755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.513720 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbnr\" (UniqueName: \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.513902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.513834 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.514639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.514615 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.582370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.580711 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbnr\" (UniqueName: \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\") pod \"payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.620558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.620510 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podStartSLOduration=2.620494333 podStartE2EDuration="2.620494333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:20.617743379 +0000 UTC m=+2235.047613241" watchObservedRunningTime="2026-11-02 23:25:20.620494333 +0000 UTC m=+2235.050364259" Nov 02 23:25:20.710226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.710164 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:20.854245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:20.846773 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:25:21.156809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.156768 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:25:21.157513 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:25:21.157482 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f3df88_897e_4562_a15a_92fc7a112d24.slice/crio-93f2f1c8c4b26b3949686651db4c1899bad2e3937b56a7839e9538ed569eabdd WatchSource:0}: Error finding container 93f2f1c8c4b26b3949686651db4c1899bad2e3937b56a7839e9538ed569eabdd: Status 404 returned error can't find the container with id 93f2f1c8c4b26b3949686651db4c1899bad2e3937b56a7839e9538ed569eabdd Nov 02 23:25:21.206871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.206850 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:21.213812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.213790 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:21.361848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.361810 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:21.362022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.361863 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:21.395838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.395748 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" event={"ID":"bea26092-fb19-476f-8f6d-df7de5c20313","Type":"ContainerStarted","Data":"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544"} Nov 02 23:25:21.395838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.395802 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" event={"ID":"bea26092-fb19-476f-8f6d-df7de5c20313","Type":"ContainerStarted","Data":"caf0e0c93c38de6c87a049839224e7ed6d1013f767ea935634aa28c6fd4feb50"} Nov 02 23:25:21.396080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.395939 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:21.397543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.397514 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" event={"ID":"43f3df88-897e-4562-a15a-92fc7a112d24","Type":"ContainerStarted","Data":"b3101c96f73dc52009ea80ba4553700dda87eb3718e722a339f201f9024d4444"} Nov 02 23:25:21.397702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.397562 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" event={"ID":"43f3df88-897e-4562-a15a-92fc7a112d24","Type":"ContainerStarted","Data":"93f2f1c8c4b26b3949686651db4c1899bad2e3937b56a7839e9538ed569eabdd"} Nov 02 23:25:21.397702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.397619 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:21.397839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.397822 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:21.398771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.398752 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:25:21.513601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.513540 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" podStartSLOduration=1.513528057 podStartE2EDuration="1.513528057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:21.512226124 +0000 UTC m=+2235.942095986" watchObservedRunningTime="2026-11-02 23:25:21.513528057 +0000 UTC m=+2235.943397951" Nov 02 23:25:21.770298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:21.770233 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" podStartSLOduration=1.7702090350000002 podStartE2EDuration="1.770209035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:21.76661745 +0000 UTC m=+2236.196487308" watchObservedRunningTime="2026-11-02 23:25:21.770209035 +0000 UTC m=+2236.200078896" Nov 02 23:25:22.361435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:22.361386 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:22.361974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:22.361469 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:23.362243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:23.362163 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 02 23:25:23.362693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:23.362273 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 02 23:25:23.386741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:23.386694 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:23.555011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:23.554939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:23.699358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:23.699330 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:24.365939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:24.365903 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:25:24.366405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:24.366208 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:25:24.366978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:24.366962 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:25:26.662640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:26.662586 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:26.855097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:26.854067 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:27.344268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:27.344213 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:28.840507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:28.840476 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:25:32.407803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:32.407773 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:25:32.408288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:32.407857 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:25:32.581385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:32.581349 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:34.073913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:34.073877 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:35.216553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:35.216516 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:35.814160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:35.814119 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:36.082506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:36.082402 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:37.961217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:37.961166 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:40.291616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:40.291576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:40.456532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:40.456504 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:40.601900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:40.601792 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:43.085733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:43.085701 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:43.574384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:43.574348 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:43.779060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:43.779021 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:47.564158 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:47.564105 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:25:47.871953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:47.871868 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:25:47.872121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:47.872071 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerName="maas-api" containerID="cri-o://22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038" gracePeriod=30 Nov 02 23:25:47.877713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:47.877676 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.169:8443/health\": dial tcp 10.133.0.169:8443: connect: connection refused" Nov 02 23:25:47.972099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:47.972063 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:48.195033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.195005 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:25:48.292198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.290592 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\") pod \"e16251e0-34f7-4572-a885-a7ba6e525ab5\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " Nov 02 23:25:48.292198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.290773 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\" (UniqueName: \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\") pod \"e16251e0-34f7-4572-a885-a7ba6e525ab5\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " Nov 02 23:25:48.292198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.290873 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\") pod \"e16251e0-34f7-4572-a885-a7ba6e525ab5\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " Nov 02 23:25:48.292198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.290941 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") pod \"e16251e0-34f7-4572-a885-a7ba6e525ab5\" (UID: \"e16251e0-34f7-4572-a885-a7ba6e525ab5\") " Nov 02 23:25:48.298234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.298189 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca" pod "e16251e0-34f7-4572-a885-a7ba6e525ab5" (UID: "e16251e0-34f7-4572-a885-a7ba6e525ab5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:48.298991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.298945 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs" pod "e16251e0-34f7-4572-a885-a7ba6e525ab5" (UID: "e16251e0-34f7-4572-a885-a7ba6e525ab5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:48.302154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.302019 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7" pod "e16251e0-34f7-4572-a885-a7ba6e525ab5" (UID: "e16251e0-34f7-4572-a885-a7ba6e525ab5"). InnerVolumeSpecName "kube-api-access-bztw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:48.304035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.303766 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls" pod "e16251e0-34f7-4572-a885-a7ba6e525ab5" (UID: "e16251e0-34f7-4572-a885-a7ba6e525ab5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:25:48.393004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.392968 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztw7\" (UniqueName: \"kubernetes.io/projected/e16251e0-34f7-4572-a885-a7ba6e525ab5-kube-api-access-bztw7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:48.393004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.393000 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:48.393004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.393013 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16251e0-34f7-4572-a885-a7ba6e525ab5-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:48.393276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.393021 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16251e0-34f7-4572-a885-a7ba6e525ab5-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:25:48.575922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.575833 4494 generic.go:356] "Generic (PLEG): container finished" podID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerID="22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038" exitCode=0 Nov 02 23:25:48.575922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.575878 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" Nov 02 23:25:48.576459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.575910 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" event={"ID":"e16251e0-34f7-4572-a885-a7ba6e525ab5","Type":"ContainerDied","Data":"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038"} Nov 02 23:25:48.576459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.575967 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp" event={"ID":"e16251e0-34f7-4572-a885-a7ba6e525ab5","Type":"ContainerDied","Data":"f1d3a8d3110b9e5903027f92fbcaddda321512dac8eb25a6236320a012455832"} Nov 02 23:25:48.576459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.575991 4494 scope.go:122] "RemoveContainer" containerID="22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038" Nov 02 23:25:48.587939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.587921 4494 scope.go:122] "RemoveContainer" containerID="22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038" Nov 02 23:25:48.588230 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:25:48.588212 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038\": container with ID starting with 22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038 not found: ID does not exist" containerID="22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038" Nov 02 23:25:48.588307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.588238 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038"} err="failed to get container status \"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038\": rpc error: code = NotFound desc = could not find container \"22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038\": container with ID starting with 22743bf94312779aa85acc2d87764edb5bb13749651d301245a9689301739038 not found: ID does not exist" Nov 02 23:25:48.744001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.743960 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:25:48.797874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:48.797831 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb3c46-c496f7894-6n9bp"] Nov 02 23:25:49.694602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:49.694567 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" path="/var/lib/kubelet/pods/e16251e0-34f7-4572-a885-a7ba6e525ab5/volumes" Nov 02 23:25:49.794457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:49.794421 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:54.734859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:54.734821 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:54.884373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:54.884331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:55.226357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:55.226323 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:56.417216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:56.417183 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:56.930166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:56.930130 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:25:56.930394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:56.930361 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" containerID="cri-o://6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19" gracePeriod=30 Nov 02 23:25:56.948156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:56.948132 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:25:57.725115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:25:57.725086 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:00.491850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:00.491771 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:00.636359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:00.636328 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:00.840820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:00.840738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:00.987788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:00.987751 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:01.145245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:01.144881 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:02.283510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.283481 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.401640 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.401708 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.401755 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.401787 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.401878 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.402021 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.402059 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.402107 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.402943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.402204 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\") pod \"841753d8-819a-4821-b6d9-df462630262b\" (UID: \"841753d8-819a-4821-b6d9-df462630262b\") " Nov 02 23:26:02.405223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.405053 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:02.406034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.405703 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:26:02.410443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.410410 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:26:02.411571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.411542 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:26:02.411709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.411613 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:26:02.414984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.414934 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:26:02.415883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.415844 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:26:02.416305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.416278 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:02.417006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.416971 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn" pod "841753d8-819a-4821-b6d9-df462630262b" (UID: "841753d8-819a-4821-b6d9-df462630262b"). InnerVolumeSpecName "kube-api-access-8vhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:02.503754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503720 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503746 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841753d8-819a-4821-b6d9-df462630262b-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503756 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhcn\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-kube-api-access-8vhcn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503767 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503777 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503787 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841753d8-819a-4821-b6d9-df462630262b-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503795 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503803 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841753d8-819a-4821-b6d9-df462630262b-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.503998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.503811 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841753d8-819a-4821-b6d9-df462630262b-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:02.678911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.678874 4494 generic.go:356] "Generic (PLEG): container finished" podID="841753d8-819a-4821-b6d9-df462630262b" containerID="6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19" exitCode=0 Nov 02 23:26:02.679091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.678936 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" Nov 02 23:26:02.679091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.678968 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerDied","Data":"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19"} Nov 02 23:26:02.679091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.679029 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22" event={"ID":"841753d8-819a-4821-b6d9-df462630262b","Type":"ContainerDied","Data":"902d29b92942b898c6337bbe8615243666c64a39879238eda11fb08a6fb13f06"} Nov 02 23:26:02.679091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.679056 4494 scope.go:122] "RemoveContainer" containerID="6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19" Nov 02 23:26:02.689597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.689581 4494 scope.go:122] "RemoveContainer" containerID="7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5" Nov 02 23:26:02.699309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.699287 4494 scope.go:122] "RemoveContainer" containerID="6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19" Nov 02 23:26:02.699593 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:02.699574 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19\": container with ID starting with 6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19 not found: ID does not exist" containerID="6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19" Nov 02 23:26:02.699681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.699601 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19"} err="failed to get container status \"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19\": rpc error: code = NotFound desc = could not find container \"6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19\": container with ID starting with 6eb917f1f47f5119d872afa3f67cdab53f9c7def165f5755cf0da5fa489f6b19 not found: ID does not exist" Nov 02 23:26:02.699681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.699621 4494 scope.go:122] "RemoveContainer" containerID="7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5" Nov 02 23:26:02.699874 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:02.699857 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5\": container with ID starting with 7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5 not found: ID does not exist" containerID="7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5" Nov 02 23:26:02.699914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.699881 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5"} err="failed to get container status \"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5\": rpc error: code = NotFound desc = could not find container \"7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5\": container with ID starting with 7ca2e9ffaf1b1b9fc926911d219db4b2bbffd63151a21774148652152ce12be5 not found: ID does not exist" Nov 02 23:26:02.846646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.846607 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:26:02.851618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:02.851593 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eb3c46-openshift-default-587b477687-lzg22"] Nov 02 23:26:03.693224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:03.693163 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841753d8-819a-4821-b6d9-df462630262b" path="/var/lib/kubelet/pods/841753d8-819a-4821-b6d9-df462630262b/volumes" Nov 02 23:26:03.816240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:03.816208 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:05.071770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:05.071736 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:05.192193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:05.192134 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:26:05.559582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:05.559543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:05.704861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:05.704828 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:05.854648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:05.854554 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:08.310080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:08.310032 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:08.518755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:08.518726 4494 scope.go:122] "RemoveContainer" containerID="c8adb2d62d9e92f9d77a691687360c12f45fa6330c65ac6e66978d7fae6ba907" Nov 02 23:26:09.438083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:09.438047 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:10.094227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.094151 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:10.193135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.193100 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" podUID="2a696e11-4df9-414e-b33b-075ec80ae374" containerName="payload-processing" containerID="cri-o://9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45" gracePeriod=25 Nov 02 23:26:10.464040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.464012 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:26:10.543755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.543714 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\" (UniqueName: \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\") pod \"2a696e11-4df9-414e-b33b-075ec80ae374\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " Nov 02 23:26:10.543963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.543889 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\") pod \"2a696e11-4df9-414e-b33b-075ec80ae374\" (UID: \"2a696e11-4df9-414e-b33b-075ec80ae374\") " Nov 02 23:26:10.544349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.544322 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume" pod "2a696e11-4df9-414e-b33b-075ec80ae374" (UID: "2a696e11-4df9-414e-b33b-075ec80ae374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:26:10.546139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.546114 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b" pod "2a696e11-4df9-414e-b33b-075ec80ae374" (UID: "2a696e11-4df9-414e-b33b-075ec80ae374"). InnerVolumeSpecName "kube-api-access-ls47b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:26:10.645251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.645112 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a696e11-4df9-414e-b33b-075ec80ae374-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:10.645251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.645162 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls47b\" (UniqueName: \"kubernetes.io/projected/2a696e11-4df9-414e-b33b-075ec80ae374-kube-api-access-ls47b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:26:10.731207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.731151 4494 generic.go:356] "Generic (PLEG): container finished" podID="2a696e11-4df9-414e-b33b-075ec80ae374" containerID="9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45" exitCode=0 Nov 02 23:26:10.731374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.731206 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" event={"ID":"2a696e11-4df9-414e-b33b-075ec80ae374","Type":"ContainerDied","Data":"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45"} Nov 02 23:26:10.731374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.731233 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" Nov 02 23:26:10.731374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.731261 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9" event={"ID":"2a696e11-4df9-414e-b33b-075ec80ae374","Type":"ContainerDied","Data":"964e9ea6c44f863ad42ecd56a6ab63e5fc5bcae91afe7c80c1c8245419b6ad49"} Nov 02 23:26:10.731374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.731284 4494 scope.go:122] "RemoveContainer" containerID="9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45" Nov 02 23:26:10.741021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.741004 4494 scope.go:122] "RemoveContainer" containerID="9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45" Nov 02 23:26:10.741347 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:10.741328 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45\": container with ID starting with 9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45 not found: ID does not exist" containerID="9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45" Nov 02 23:26:10.741436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.741355 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45"} err="failed to get container status \"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45\": rpc error: code = NotFound desc = could not find container \"9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45\": container with ID starting with 9f71b01dd0310d217dfa0e9f9f8280bd092be1420d07f6e82e4ba167567e2d45 not found: ID does not exist" Nov 02 23:26:10.900945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.900858 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:26:10.904973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:10.904943 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e875bc-789b45c54f-59sq9"] Nov 02 23:26:11.189452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:11.189421 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:11.695011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:11.694981 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a696e11-4df9-414e-b33b-075ec80ae374" path="/var/lib/kubelet/pods/2a696e11-4df9-414e-b33b-075ec80ae374/volumes" Nov 02 23:26:14.487621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:14.487592 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:26:16.104645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.104611 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:26:16.105263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105248 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2a696e11-4df9-414e-b33b-075ec80ae374" containerName="payload-processing" Nov 02 23:26:16.105263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105265 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a696e11-4df9-414e-b33b-075ec80ae374" containerName="payload-processing" Nov 02 23:26:16.105364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105282 4494 cpu_manager.go:414] "containerMap: removing container" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:16.105364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105289 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:16.105364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105335 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerName="maas-api" Nov 02 23:26:16.105364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105344 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerName="maas-api" Nov 02 23:26:16.105492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105365 4494 cpu_manager.go:414] "containerMap: removing container" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:16.105492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105373 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:16.105555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105508 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a696e11-4df9-414e-b33b-075ec80ae374" containerName="payload-processing" Nov 02 23:26:16.105555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105521 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:16.105555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.105532 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16251e0-34f7-4572-a885-a7ba6e525ab5" containerName="maas-api" Nov 02 23:26:16.110348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.110326 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.163384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.163352 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d530c9-serving-cert\"" Nov 02 23:26:16.210596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.210560 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.210791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.210690 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.210871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.210854 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscb5\" (UniqueName: \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.210919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.210883 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.216698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.216674 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:26:16.296531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.296497 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:16.312038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.312011 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscb5\" (UniqueName: \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.312207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.312047 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.312207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.312110 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.312207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.312145 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.312341 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:16.312313 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-serving-cert: secret "maas-api-e2e-praxis-a-d530c9-serving-cert" not found Nov 02 23:26:16.312395 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:16.312384 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls podName:69578c7c-0d3c-4cc4-ab99-76be50f0cb36 nodeName:}" failed. No retries permitted until 2026-11-02 23:26:16.812369385 +0000 UTC m=+2291.242239228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls") pod "maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" (UID: "69578c7c-0d3c-4cc4-ab99-76be50f0cb36") : secret "maas-api-e2e-praxis-a-d530c9-serving-cert" not found Nov 02 23:26:16.312810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.312793 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.314712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.314696 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.376660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.376592 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscb5\" (UniqueName: \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.464462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.464425 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:16.609076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.609018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:16.820698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.820661 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:16.823211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:16.823187 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:17.023281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:17.023243 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:17.401827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:17.401803 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:26:17.402940 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:26:17.402919 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69578c7c_0d3c_4cc4_ab99_76be50f0cb36.slice/crio-80f2cb8797f54ab05a04f80855096b5ac309e1f1c21841d2fd8d80776466874c WatchSource:0}: Error finding container 80f2cb8797f54ab05a04f80855096b5ac309e1f1c21841d2fd8d80776466874c: Status 404 returned error can't find the container with id 80f2cb8797f54ab05a04f80855096b5ac309e1f1c21841d2fd8d80776466874c Nov 02 23:26:17.560775 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:17.560686 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:26:17.783840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:17.783798 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" event={"ID":"69578c7c-0d3c-4cc4-ab99-76be50f0cb36","Type":"ContainerStarted","Data":"80f2cb8797f54ab05a04f80855096b5ac309e1f1c21841d2fd8d80776466874c"} Nov 02 23:26:18.791471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:18.791437 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" event={"ID":"69578c7c-0d3c-4cc4-ab99-76be50f0cb36","Type":"ContainerStarted","Data":"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41"} Nov 02 23:26:18.791963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:18.791698 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:18.916016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:18.915955 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" podStartSLOduration=1.93147215 podStartE2EDuration="2.915941606s" totalImagesPullingTime="984.469456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:18.914313978 +0000 UTC m=+2293.344183848" watchObservedRunningTime="2026-11-02 23:26:18.915941606 +0000 UTC m=+2293.345811466" Nov 02 23:26:21.428483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:21.428444 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:24.803120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:24.803085 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:26:26.644946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:26.644903 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:32.329462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.329426 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:32.477448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.477412 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:32.785698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.785665 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:26:32.786501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.786481 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="841753d8-819a-4821-b6d9-df462630262b" containerName="istio-proxy" Nov 02 23:26:32.789855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.789839 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.819216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.819157 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvnd\" (UniqueName: \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.819350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.819336 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.819479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.819462 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.819570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.819553 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.844622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.844597 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a2f549-serving-cert\"" Nov 02 23:26:32.898248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.898218 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:26:32.920591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.920560 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvnd\" (UniqueName: \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.920852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.920663 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.920852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.920761 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.920852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.920845 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.921054 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:32.920983 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-serving-cert: secret "maas-api-e2e-praxis-b-a2f549-serving-cert" not found Nov 02 23:26:32.921054 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:32.921049 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls podName:acdab9f9-9108-4064-a6bd-8a24587d3d8e nodeName:}" failed. No retries permitted until 2026-11-02 23:26:33.42102903 +0000 UTC m=+2307.850898888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls") pod "maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" (UID: "acdab9f9-9108-4064-a6bd-8a24587d3d8e") : secret "maas-api-e2e-praxis-b-a2f549-serving-cert" not found Nov 02 23:26:32.922036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.921990 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.923983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.923958 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.982832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.982800 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvnd\" (UniqueName: \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:32.984564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:32.984539 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:33.139731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:33.139643 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:33.287200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:33.287143 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:33.427974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:33.427947 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:33.430536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:33.430516 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:33.703256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:33.703158 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:34.085948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:34.085915 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:26:34.087828 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:26:34.087798 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdab9f9_9108_4064_a6bd_8a24587d3d8e.slice/crio-7f6da53b244e0f839b83544d59f1cd5ac3388fe330d7309ad77ee643645cf50e WatchSource:0}: Error finding container 7f6da53b244e0f839b83544d59f1cd5ac3388fe330d7309ad77ee643645cf50e: Status 404 returned error can't find the container with id 7f6da53b244e0f839b83544d59f1cd5ac3388fe330d7309ad77ee643645cf50e Nov 02 23:26:34.889777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:34.889740 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" event={"ID":"acdab9f9-9108-4064-a6bd-8a24587d3d8e","Type":"ContainerStarted","Data":"7f6da53b244e0f839b83544d59f1cd5ac3388fe330d7309ad77ee643645cf50e"} Nov 02 23:26:35.897270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:35.897229 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" event={"ID":"acdab9f9-9108-4064-a6bd-8a24587d3d8e","Type":"ContainerStarted","Data":"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7"} Nov 02 23:26:35.897740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:35.897333 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:36.017095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:36.017036 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" podStartSLOduration=3.167251855 podStartE2EDuration="4.017021336s" totalImagesPullingTime="849.769481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:36.01629733 +0000 UTC m=+2310.446167204" watchObservedRunningTime="2026-11-02 23:26:36.017021336 +0000 UTC m=+2310.446891196" Nov 02 23:26:38.064649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:38.064614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:41.909280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:41.909249 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:26:42.985011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:42.984968 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:47.564701 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:26:47.564637 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:26:47.935591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:47.935546 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:49.032011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.031970 4494 generic.go:356] "Generic (PLEG): container finished" podID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerID="edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca" exitCode=0 Nov 02 23:26:49.032499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.032047 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerDied","Data":"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca"} Nov 02 23:26:49.032549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.032541 4494 scope.go:122] "RemoveContainer" containerID="edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca" Nov 02 23:26:49.034222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.034195 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:26:49.034301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.034264 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:26:49.034301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:49.034295 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:26:50.059518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:50.059485 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerStarted","Data":"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19"} Nov 02 23:26:50.977894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:50.977862 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:26:50.978124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:50.978105 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:26:50.978215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:50.978147 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 23:26:51.978830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:51.978797 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:26:51.979346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:51.978855 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 23:26:52.750715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:52.750680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:52.901893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:52.901844 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:52.983469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:52.983432 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:26:52.984103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:52.984067 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 23:26:53.057204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:53.057061 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:26:53.978275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:53.978240 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:26:53.978461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:53.978296 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 23:26:54.978648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:54.978609 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 23:26:54.979064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:54.978667 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 23:26:55.982137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:55.982093 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:26:55.982581 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:55.982479 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:26:55.983092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:55.983077 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:26:56.081752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:56.081719 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:26:57.889016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:26:57.888979 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:03.107465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:03.107413 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:06.202603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.202570 4494 generic.go:356] "Generic (PLEG): container finished" podID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerID="dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9" exitCode=0 Nov 02 23:27:06.203094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.202639 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerDied","Data":"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9"} Nov 02 23:27:06.203094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.203060 4494 scope.go:122] "RemoveContainer" containerID="dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9" Nov 02 23:27:06.204706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.204677 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:06.204810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.204743 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:06.204810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.204776 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:06.397889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.397799 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:27:06.621770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:06.621735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:07.113486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:07.113453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:07.232983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:07.232945 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerStarted","Data":"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e"} Nov 02 23:27:08.207088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.207053 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:27:08.208935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.208901 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 23:27:08.209108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.208964 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 23:27:08.211308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.211286 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:27:08.215422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.215407 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.272378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.272344 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:27:08.296232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.296157 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.296475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.296443 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh67p\" (UniqueName: \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.319389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.319352 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-014ab4-dockercfg-48nml\"" Nov 02 23:27:08.319389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.319352 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-014ab4\"" Nov 02 23:27:08.398165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.398130 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.398420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.398243 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh67p\" (UniqueName: \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.398967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.398938 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.460396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.460309 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh67p\" (UniqueName: \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\") pod \"payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.529416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.529382 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:08.659828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.659771 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:08.961226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:08.960915 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:27:08.962505 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:27:08.962472 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a02255_0c0e_4293_99a4_fc60d4321022.slice/crio-6dae6d511a489b6e6a5386ee6c5d4bdbc351b45c98e65837967b1abfe78ae2aa WatchSource:0}: Error finding container 6dae6d511a489b6e6a5386ee6c5d4bdbc351b45c98e65837967b1abfe78ae2aa: Status 404 returned error can't find the container with id 6dae6d511a489b6e6a5386ee6c5d4bdbc351b45c98e65837967b1abfe78ae2aa Nov 02 23:27:09.002056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.002023 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:27:09.007219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.007192 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.065153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.065126 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:27:09.108190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.108139 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbnn\" (UniqueName: \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.108468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.108326 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.207439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.207398 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 23:27:09.207703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.207456 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 23:27:09.210504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.210431 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbnn\" (UniqueName: \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.210611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.210586 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.211402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.211382 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.272439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.272403 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbnn\" (UniqueName: \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\") pod \"payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.279850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.279813 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" event={"ID":"25a02255-0c0e-4293-99a4-fc60d4321022","Type":"ContainerStarted","Data":"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd"} Nov 02 23:27:09.280016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.279880 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" event={"ID":"25a02255-0c0e-4293-99a4-fc60d4321022","Type":"ContainerStarted","Data":"6dae6d511a489b6e6a5386ee6c5d4bdbc351b45c98e65837967b1abfe78ae2aa"} Nov 02 23:27:09.280114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.280091 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:09.284689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.284665 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:09.363924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.363884 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:09.435675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.435544 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" podStartSLOduration=1.435521603 podStartE2EDuration="1.435521603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:09.432655489 +0000 UTC m=+2343.862525367" watchObservedRunningTime="2026-11-02 23:27:09.435521603 +0000 UTC m=+2343.865391483" Nov 02 23:27:09.823189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:09.823146 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:27:09.824264 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:27:09.824240 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b131e8_bada_4765_967e_3d20c22c4d96.slice/crio-fb4e0e83bfd3c8491c26473c77e88f9c85ebaf04d9fbe47cf790944344547441 WatchSource:0}: Error finding container fb4e0e83bfd3c8491c26473c77e88f9c85ebaf04d9fbe47cf790944344547441: Status 404 returned error can't find the container with id fb4e0e83bfd3c8491c26473c77e88f9c85ebaf04d9fbe47cf790944344547441 Nov 02 23:27:10.212028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.211579 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 23:27:10.212028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.211638 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 23:27:10.295220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.292649 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" event={"ID":"34b131e8-bada-4765-967e-3d20c22c4d96","Type":"ContainerStarted","Data":"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d"} Nov 02 23:27:10.295220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.292720 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" event={"ID":"34b131e8-bada-4765-967e-3d20c22c4d96","Type":"ContainerStarted","Data":"fb4e0e83bfd3c8491c26473c77e88f9c85ebaf04d9fbe47cf790944344547441"} Nov 02 23:27:10.295220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.292880 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:10.413194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:10.413107 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" podStartSLOduration=2.4130906899999998 podStartE2EDuration="2.41309069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:10.410961281 +0000 UTC m=+2344.840831142" watchObservedRunningTime="2026-11-02 23:27:10.41309069 +0000 UTC m=+2344.842960557" Nov 02 23:27:11.087038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:11.087000 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:11.207795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:11.207756 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 23:27:11.207980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:11.207826 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 23:27:12.207713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:12.207674 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 02 23:27:12.208254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:12.207731 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 02 23:27:13.221447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:13.221408 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:27:13.222014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:13.221839 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:27:13.223579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:13.223553 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:27:16.057346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:16.057306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:17.512097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:17.512041 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:17.592095 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:27:17.592046 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:27:17.658644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:17.658609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:19.122912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:19.122882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:19.658293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:19.658257 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:27:19.658606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:19.658480 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" containerID="cri-o://0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1" gracePeriod=30 Nov 02 23:27:19.659870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:19.659813 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:20.294845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:20.294814 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:27:20.603279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:20.603153 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:21.199635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:21.199587 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:21.321264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:21.320370 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:27:21.347725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:21.347693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:21.492835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:21.492730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:24.956203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:24.956159 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:27:25.046715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046682 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.046891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046761 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.046891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046783 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.046891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046808 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.046891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046882 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.047098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046914 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.047098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046944 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.047098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.046983 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.047098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047006 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\") pod \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\" (UID: \"38368803-a2bb-44c3-b4bd-e90f86f56b8a\") " Nov 02 23:27:25.047432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047054 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:25.047432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047305 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:25.047432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047321 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:25.047780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047684 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:25.047780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047707 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.047780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047733 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.047780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047749 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.047780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.047747 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:25.049647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.049621 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:25.049750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.049670 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:25.049750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.049726 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:27:25.049841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.049821 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m" pod "38368803-a2bb-44c3-b4bd-e90f86f56b8a" (UID: "38368803-a2bb-44c3-b4bd-e90f86f56b8a"). InnerVolumeSpecName "kube-api-access-swn5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:25.149297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149267 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn5m\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-kube-api-access-swn5m\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.149297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149297 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.149487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149308 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38368803-a2bb-44c3-b4bd-e90f86f56b8a-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.149487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149321 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.149487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149331 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.149487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.149339 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38368803-a2bb-44c3-b4bd-e90f86f56b8a-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:27:25.402807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.402710 4494 generic.go:356] "Generic (PLEG): container finished" podID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerID="0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1" exitCode=0 Nov 02 23:27:25.402807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.402753 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" Nov 02 23:27:25.403013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.402803 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerDied","Data":"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1"} Nov 02 23:27:25.403013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.402862 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv" event={"ID":"38368803-a2bb-44c3-b4bd-e90f86f56b8a","Type":"ContainerDied","Data":"6c5c7f593d26d8aec661f4fa7abf07e2286c1282989ae80fb8ba624c6a55cb5a"} Nov 02 23:27:25.403013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.402891 4494 scope.go:122] "RemoveContainer" containerID="0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1" Nov 02 23:27:25.414840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.414814 4494 scope.go:122] "RemoveContainer" containerID="5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b" Nov 02 23:27:25.424738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.424718 4494 scope.go:122] "RemoveContainer" containerID="0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1" Nov 02 23:27:25.425043 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:27:25.425021 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1\": container with ID starting with 0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1 not found: ID does not exist" containerID="0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1" Nov 02 23:27:25.425104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.425051 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1"} err="failed to get container status \"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1\": rpc error: code = NotFound desc = could not find container \"0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1\": container with ID starting with 0ef0761f64b4ef6f20a9173bfef4eeca013f0b9e7897ce4dff1fc7d2bc10f9b1 not found: ID does not exist" Nov 02 23:27:25.425104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.425071 4494 scope.go:122] "RemoveContainer" containerID="5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b" Nov 02 23:27:25.425355 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:27:25.425341 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b\": container with ID starting with 5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b not found: ID does not exist" containerID="5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b" Nov 02 23:27:25.425407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.425358 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b"} err="failed to get container status \"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b\": rpc error: code = NotFound desc = could not find container \"5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b\": container with ID starting with 5923b52e03edf31a09e528dceb311263a9eaa67a07c2ebf2cd92c893f81f677b not found: ID does not exist" Nov 02 23:27:25.570437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.570407 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:27:25.620831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.620803 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e875bc-openshift-default-595c678ff-kttqv"] Nov 02 23:27:25.695887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.695851 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" path="/var/lib/kubelet/pods/38368803-a2bb-44c3-b4bd-e90f86f56b8a/volumes" Nov 02 23:27:25.850071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:25.850031 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:26.001934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:26.001823 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:26.148898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:26.148859 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:26.568159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:26.568128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:27.952999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:27.952968 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv_34b131e8-bada-4765-967e-3d20c22c4d96/payload-processing/0.log" Nov 02 23:27:29.090707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:29.090678 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:29.266359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:29.266318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:31.707152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:31.707122 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:32.414343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:32.414296 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:33.147595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:33.147563 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:34.199258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.199225 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:27:34.330125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.330091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:34.865505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.865473 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:27:34.866099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.866084 4494 cpu_manager.go:414] "containerMap: removing container" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:27:34.866156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.866100 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:27:34.866156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.866112 4494 cpu_manager.go:414] "containerMap: removing container" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:27:34.866156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.866117 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:27:34.866554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.866225 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:27:34.869842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.869822 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891639 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891710 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891787 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891820 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891851 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891894 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.891953 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfr29\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.892009 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.900819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.892034 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:34.923197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.923115 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-adfeab-openshift-default-dockercfg-7qfkq\"" Nov 02 23:27:34.955734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.955669 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:34.981795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:34.981750 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:27:35.006097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.006064 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.006395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.006369 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016713 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016775 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016824 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016881 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfr29\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016944 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.016972 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.017141 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.017214 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.017636 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.018205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.019098 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.019355 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.049625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.036693 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.065521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.065490 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.095584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.095547 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfr29\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.098892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.098862 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\") pod \"e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.183575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.183542 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:35.264118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.262692 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:35.674568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.674544 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:27:35.675748 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:27:35.675721 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e41bd5_3d46_4b8d_9623_709439954870.slice/crio-4b4936b0554cbd6be096dc46833d45c879e822896f91181b5e71737b103d58a7 WatchSource:0}: Error finding container 4b4936b0554cbd6be096dc46833d45c879e822896f91181b5e71737b103d58a7: Status 404 returned error can't find the container with id 4b4936b0554cbd6be096dc46833d45c879e822896f91181b5e71737b103d58a7 Nov 02 23:27:35.678270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.678243 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:35.678333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.678303 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:35.678375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:35.678332 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:27:36.523824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:36.523788 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerStarted","Data":"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc"} Nov 02 23:27:36.524251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:36.523854 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerStarted","Data":"4b4936b0554cbd6be096dc46833d45c879e822896f91181b5e71737b103d58a7"} Nov 02 23:27:36.647655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:36.647601 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podStartSLOduration=2.647588383 podStartE2EDuration="2.647588383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:36.644621962 +0000 UTC m=+2371.074491825" watchObservedRunningTime="2026-11-02 23:27:36.647588383 +0000 UTC m=+2371.077458240" Nov 02 23:27:37.184085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:37.184053 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:37.189786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:37.189764 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:37.349594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:37.349558 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:37.530212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:37.530102 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:37.530991 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:37.530972 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:27:39.890613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:39.890578 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:40.057104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:40.057064 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:40.210023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:40.209991 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:43.101642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:43.101608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:43.299600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:43.299567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:44.409546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:44.409511 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:47.561412 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:27:47.561330 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:27:49.743942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:49.743902 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:51.623927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:51.623890 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:54.631151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:54.631121 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:55.256553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:55.256514 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:27:59.090522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:27:59.090485 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:00.707478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:00.707443 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:00.942520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:00.942489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:01.866562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:01.866529 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:28:01.979601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:01.979558 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:28:02.226800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:02.226772 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:02.373309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:02.373275 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:06.090093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:06.090059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:06.597782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:06.597748 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:06.867767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:06.867656 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" podUID="0c23b809-0a23-4e88-8c66-7ff20ca80376" containerName="payload-processing" containerID="cri-o://7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf" gracePeriod=25 Nov 02 23:28:06.979930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:06.979888 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" podUID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" containerName="payload-pre-processing" containerID="cri-o://fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6" gracePeriod=25 Nov 02 23:28:07.140459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.140426 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:28:07.177709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.177678 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\") pod \"0c23b809-0a23-4e88-8c66-7ff20ca80376\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " Nov 02 23:28:07.177889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.177825 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\" (UniqueName: \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\") pod \"0c23b809-0a23-4e88-8c66-7ff20ca80376\" (UID: \"0c23b809-0a23-4e88-8c66-7ff20ca80376\") " Nov 02 23:28:07.178237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.178206 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume" pod "0c23b809-0a23-4e88-8c66-7ff20ca80376" (UID: "0c23b809-0a23-4e88-8c66-7ff20ca80376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:07.181485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.181463 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk" pod "0c23b809-0a23-4e88-8c66-7ff20ca80376" (UID: "0c23b809-0a23-4e88-8c66-7ff20ca80376"). InnerVolumeSpecName "kube-api-access-4jhsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:07.240939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.240906 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:28:07.279622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.279586 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\") pod \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " Nov 02 23:28:07.279822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.279631 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\" (UniqueName: \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\") pod \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\" (UID: \"a3b54e5a-571d-48f6-bc06-63c4fba211f0\") " Nov 02 23:28:07.280077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.280045 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume" pod "a3b54e5a-571d-48f6-bc06-63c4fba211f0" (UID: "a3b54e5a-571d-48f6-bc06-63c4fba211f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:07.280239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.280221 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c23b809-0a23-4e88-8c66-7ff20ca80376-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:07.280305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.280247 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jhsk\" (UniqueName: \"kubernetes.io/projected/0c23b809-0a23-4e88-8c66-7ff20ca80376-kube-api-access-4jhsk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:07.280305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.280264 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b54e5a-571d-48f6-bc06-63c4fba211f0-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:07.281988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.281960 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv" pod "a3b54e5a-571d-48f6-bc06-63c4fba211f0" (UID: "a3b54e5a-571d-48f6-bc06-63c4fba211f0"). InnerVolumeSpecName "kube-api-access-pttcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:07.381494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.381419 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttcv\" (UniqueName: \"kubernetes.io/projected/a3b54e5a-571d-48f6-bc06-63c4fba211f0-kube-api-access-pttcv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:07.720690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.720659 4494 generic.go:356] "Generic (PLEG): container finished" podID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" containerID="fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6" exitCode=0 Nov 02 23:28:07.720867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.720699 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" Nov 02 23:28:07.720867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.720738 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" event={"ID":"a3b54e5a-571d-48f6-bc06-63c4fba211f0","Type":"ContainerDied","Data":"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6"} Nov 02 23:28:07.720867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.720786 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8" event={"ID":"a3b54e5a-571d-48f6-bc06-63c4fba211f0","Type":"ContainerDied","Data":"367a1f0ed514f4535e8c92d70ac3c789c3865e44797ac1b7e77065464a94d1bc"} Nov 02 23:28:07.720867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.720813 4494 scope.go:122] "RemoveContainer" containerID="fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6" Nov 02 23:28:07.722155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.722128 4494 generic.go:356] "Generic (PLEG): container finished" podID="0c23b809-0a23-4e88-8c66-7ff20ca80376" containerID="7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf" exitCode=0 Nov 02 23:28:07.722317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.722165 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" Nov 02 23:28:07.722317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.722209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" event={"ID":"0c23b809-0a23-4e88-8c66-7ff20ca80376","Type":"ContainerDied","Data":"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf"} Nov 02 23:28:07.722317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.722256 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j" event={"ID":"0c23b809-0a23-4e88-8c66-7ff20ca80376","Type":"ContainerDied","Data":"37287da17557aa47e3d076bf7a5063380545310533d9f62928b67978fdda8dd4"} Nov 02 23:28:07.731363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.731337 4494 scope.go:122] "RemoveContainer" containerID="fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6" Nov 02 23:28:07.731654 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:07.731634 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6\": container with ID starting with fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6 not found: ID does not exist" containerID="fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6" Nov 02 23:28:07.731714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.731695 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6"} err="failed to get container status \"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6\": rpc error: code = NotFound desc = could not find container \"fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6\": container with ID starting with fe3684ef7ae847d44c76aa28bfca1ef8c41c75c908a2fb574b62996cad7942c6 not found: ID does not exist" Nov 02 23:28:07.731766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.731714 4494 scope.go:122] "RemoveContainer" containerID="7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf" Nov 02 23:28:07.741255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.741235 4494 scope.go:122] "RemoveContainer" containerID="7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf" Nov 02 23:28:07.741536 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:07.741515 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf\": container with ID starting with 7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf not found: ID does not exist" containerID="7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf" Nov 02 23:28:07.741609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.741544 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf"} err="failed to get container status \"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf\": rpc error: code = NotFound desc = could not find container \"7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf\": container with ID starting with 7c2512318703739aa4917f33a7890c46d9b0644adb003bc321369d205025a5bf not found: ID does not exist" Nov 02 23:28:07.888378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.888349 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:28:07.894600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:07.894576 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-03fe6e-6876cd446c-xsvr8"] Nov 02 23:28:08.005144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:08.005018 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:28:08.009034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:08.009002 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-03fe6e-79769fff89-bwl8j"] Nov 02 23:28:09.036481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:09.036447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:09.694521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:09.694486 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c23b809-0a23-4e88-8c66-7ff20ca80376" path="/var/lib/kubelet/pods/0c23b809-0a23-4e88-8c66-7ff20ca80376/volumes" Nov 02 23:28:09.694923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:09.694910 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" path="/var/lib/kubelet/pods/a3b54e5a-571d-48f6-bc06-63c4fba211f0/volumes" Nov 02 23:28:11.138522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:11.138468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:13.462285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:13.462254 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:16.376520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:16.376487 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:28:16.626863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:16.626757 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:17.559984 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:17.559947 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:28:17.678470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:17.678430 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:23.184553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:23.184518 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:23.708778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:23.708748 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:24.444841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:24.444811 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:28:24.557119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:24.557088 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:28:24.806492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:24.806407 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:24.953228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:24.953196 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:28.546598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:28.546565 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:29.400968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.400913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:29.445432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.445393 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" podUID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" containerName="payload-processing" containerID="cri-o://18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7" gracePeriod=25 Nov 02 23:28:29.558335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.558299 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" podUID="80139230-5c41-4b6e-9e82-f9f49d67bda0" containerName="payload-pre-processing" containerID="cri-o://b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30" gracePeriod=25 Nov 02 23:28:29.733224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.733199 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:28:29.835501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.835473 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:28:29.864276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.864249 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\" (UniqueName: \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\") pod \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " Nov 02 23:28:29.864543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.864521 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\") pod \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\" (UID: \"554fe342-5956-41cb-bf7d-00fd24ee6ffd\") " Nov 02 23:28:29.864973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.864942 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume" pod "554fe342-5956-41cb-bf7d-00fd24ee6ffd" (UID: "554fe342-5956-41cb-bf7d-00fd24ee6ffd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:29.867035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.867005 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd" pod "554fe342-5956-41cb-bf7d-00fd24ee6ffd" (UID: "554fe342-5956-41cb-bf7d-00fd24ee6ffd"). InnerVolumeSpecName "kube-api-access-wnpxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:29.899227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.899194 4494 generic.go:356] "Generic (PLEG): container finished" podID="80139230-5c41-4b6e-9e82-f9f49d67bda0" containerID="b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30" exitCode=0 Nov 02 23:28:29.899345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.899256 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" Nov 02 23:28:29.899345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.899263 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" event={"ID":"80139230-5c41-4b6e-9e82-f9f49d67bda0","Type":"ContainerDied","Data":"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30"} Nov 02 23:28:29.899345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.899322 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz" event={"ID":"80139230-5c41-4b6e-9e82-f9f49d67bda0","Type":"ContainerDied","Data":"87dc156a9876397f42530f616fa3ea55004a139dc5944ed8e60eb2d80cbdde01"} Nov 02 23:28:29.899345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.899344 4494 scope.go:122] "RemoveContainer" containerID="b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30" Nov 02 23:28:29.900923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.900902 4494 generic.go:356] "Generic (PLEG): container finished" podID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" containerID="18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7" exitCode=0 Nov 02 23:28:29.901046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.900943 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" Nov 02 23:28:29.901046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.900974 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" event={"ID":"554fe342-5956-41cb-bf7d-00fd24ee6ffd","Type":"ContainerDied","Data":"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7"} Nov 02 23:28:29.901046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.901015 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9" event={"ID":"554fe342-5956-41cb-bf7d-00fd24ee6ffd","Type":"ContainerDied","Data":"124557986b607ce26f775051ef8eda30a0c767bf90d799c6640887b18b9a67f1"} Nov 02 23:28:29.909522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.909507 4494 scope.go:122] "RemoveContainer" containerID="b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30" Nov 02 23:28:29.909801 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:29.909781 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30\": container with ID starting with b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30 not found: ID does not exist" containerID="b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30" Nov 02 23:28:29.909875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.909810 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30"} err="failed to get container status \"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30\": rpc error: code = NotFound desc = could not find container \"b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30\": container with ID starting with b862466cacf5dc1db49da20ea846e69ccfc771d1d7b4710f2f14a180bb302d30 not found: ID does not exist" Nov 02 23:28:29.909875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.909826 4494 scope.go:122] "RemoveContainer" containerID="18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7" Nov 02 23:28:29.918892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.918873 4494 scope.go:122] "RemoveContainer" containerID="18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7" Nov 02 23:28:29.919144 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:29.919124 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7\": container with ID starting with 18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7 not found: ID does not exist" containerID="18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7" Nov 02 23:28:29.919227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.919152 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7"} err="failed to get container status \"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7\": rpc error: code = NotFound desc = could not find container \"18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7\": container with ID starting with 18b87a09a659355e66122d3c1dc91552a13385a6e0202d18708d396f2d6f8ea7 not found: ID does not exist" Nov 02 23:28:29.965984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.965906 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\" (UniqueName: \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\") pod \"80139230-5c41-4b6e-9e82-f9f49d67bda0\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " Nov 02 23:28:29.966110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.966073 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\") pod \"80139230-5c41-4b6e-9e82-f9f49d67bda0\" (UID: \"80139230-5c41-4b6e-9e82-f9f49d67bda0\") " Nov 02 23:28:29.966518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.966496 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume" pod "80139230-5c41-4b6e-9e82-f9f49d67bda0" (UID: "80139230-5c41-4b6e-9e82-f9f49d67bda0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:29.966660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.966647 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/554fe342-5956-41cb-bf7d-00fd24ee6ffd-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:29.966705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.966663 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpxd\" (UniqueName: \"kubernetes.io/projected/554fe342-5956-41cb-bf7d-00fd24ee6ffd-kube-api-access-wnpxd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:29.966705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.966671 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80139230-5c41-4b6e-9e82-f9f49d67bda0-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:29.968608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:29.968581 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft" pod "80139230-5c41-4b6e-9e82-f9f49d67bda0" (UID: "80139230-5c41-4b6e-9e82-f9f49d67bda0"). InnerVolumeSpecName "kube-api-access-289ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:30.068030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:30.067990 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289ft\" (UniqueName: \"kubernetes.io/projected/80139230-5c41-4b6e-9e82-f9f49d67bda0-kube-api-access-289ft\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:30.135024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:30.134993 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:28:30.138733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:30.138702 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-11f7a8-5c4dd55dff-phxj9"] Nov 02 23:28:30.321387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:30.321269 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:28:30.325290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:30.325261 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-11f7a8-7f568f59d7-cg4cz"] Nov 02 23:28:31.698436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:31.698399 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" path="/var/lib/kubelet/pods/554fe342-5956-41cb-bf7d-00fd24ee6ffd/volumes" Nov 02 23:28:31.699688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:31.699663 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80139230-5c41-4b6e-9e82-f9f49d67bda0" path="/var/lib/kubelet/pods/80139230-5c41-4b6e-9e82-f9f49d67bda0/volumes" Nov 02 23:28:32.851517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:32.851484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:34.295020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.294987 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:28:34.295601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295587 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0c23b809-0a23-4e88-8c66-7ff20ca80376" containerName="payload-processing" Nov 02 23:28:34.295660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295602 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c23b809-0a23-4e88-8c66-7ff20ca80376" containerName="payload-processing" Nov 02 23:28:34.295660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295618 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" containerName="payload-pre-processing" Nov 02 23:28:34.295660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295623 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" containerName="payload-pre-processing" Nov 02 23:28:34.295660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295632 4494 cpu_manager.go:414] "containerMap: removing container" podUID="80139230-5c41-4b6e-9e82-f9f49d67bda0" containerName="payload-pre-processing" Nov 02 23:28:34.295660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295637 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80139230-5c41-4b6e-9e82-f9f49d67bda0" containerName="payload-pre-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295673 4494 cpu_manager.go:414] "containerMap: removing container" podUID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" containerName="payload-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295677 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" containerName="payload-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295756 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b54e5a-571d-48f6-bc06-63c4fba211f0" containerName="payload-pre-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295764 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="80139230-5c41-4b6e-9e82-f9f49d67bda0" containerName="payload-pre-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295772 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="38368803-a2bb-44c3-b4bd-e90f86f56b8a" containerName="istio-proxy" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295783 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="554fe342-5956-41cb-bf7d-00fd24ee6ffd" containerName="payload-processing" Nov 02 23:28:34.295858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.295792 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c23b809-0a23-4e88-8c66-7ff20ca80376" containerName="payload-processing" Nov 02 23:28:34.299093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.299075 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.360256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.360214 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.360476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.360334 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr2f\" (UniqueName: \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.360476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.360371 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.360611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.360542 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.404107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.404073 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-818fe8-serving-cert\"" Nov 02 23:28:34.407157 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.407120 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:28:34.461412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.461381 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.461604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.461449 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr2f\" (UniqueName: \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.461604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.461482 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.461604 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:34.461543 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-serving-cert: secret "maas-api-e2e-ipp-a-818fe8-serving-cert" not found Nov 02 23:28:34.461604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.461587 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.461810 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:34.461620 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls podName:2825a71f-3473-4646-878a-2af4f3c27c22 nodeName:}" failed. No retries permitted until 2026-11-02 23:28:34.961598975 +0000 UTC m=+2429.391468827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls") pod "maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" (UID: "2825a71f-3473-4646-878a-2af4f3c27c22") : secret "maas-api-e2e-ipp-a-818fe8-serving-cert" not found Nov 02 23:28:34.462228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.462205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.464098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.464080 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.492486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.492456 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:34.526021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.525991 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr2f\" (UniqueName: \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.651112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.651025 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:34.796781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.796749 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:34.967961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.967926 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:34.970557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:34.970527 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:35.213069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:35.213031 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:35.610688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:35.610658 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:28:35.611866 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:28:35.611842 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2825a71f_3473_4646_878a_2af4f3c27c22.slice/crio-f87666b7d9fcf21faabd3b688c1ab4b14401078ac2f3f337fc65be8c13ea7ed1 WatchSource:0}: Error finding container f87666b7d9fcf21faabd3b688c1ab4b14401078ac2f3f337fc65be8c13ea7ed1: Status 404 returned error can't find the container with id f87666b7d9fcf21faabd3b688c1ab4b14401078ac2f3f337fc65be8c13ea7ed1 Nov 02 23:28:35.946806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:35.946772 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" event={"ID":"2825a71f-3473-4646-878a-2af4f3c27c22","Type":"ContainerStarted","Data":"f87666b7d9fcf21faabd3b688c1ab4b14401078ac2f3f337fc65be8c13ea7ed1"} Nov 02 23:28:36.955829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:36.955791 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" event={"ID":"2825a71f-3473-4646-878a-2af4f3c27c22","Type":"ContainerStarted","Data":"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d"} Nov 02 23:28:36.956287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:36.955871 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:37.075642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:37.075589 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" podStartSLOduration=2.196295994 podStartE2EDuration="3.075575534s" totalImagesPullingTime="879.27954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:37.074103371 +0000 UTC m=+2431.503973253" watchObservedRunningTime="2026-11-02 23:28:37.075575534 +0000 UTC m=+2431.505445394" Nov 02 23:28:38.757913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:38.757856 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:38.903631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:38.903596 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:39.729012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:39.728976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:40.723063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:40.723034 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:41.214508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:41.214474 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:28:41.214689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:41.214670 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" containerID="cri-o://45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508" gracePeriod=30 Nov 02 23:28:41.217785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:41.217756 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:42.968496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:42.968468 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:28:45.088691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:45.088657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:46.490668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.490613 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:28:46.526043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.526008 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.661646 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.661738 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.661865 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.661950 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.661992 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662053 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662251 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662306 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.662458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662411 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:46.663198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662667 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.663198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662933 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.663198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.662993 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\") pod \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\" (UID: \"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4\") " Nov 02 23:28:46.663198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663090 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.663198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663146 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.663632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663547 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.663632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663570 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.663632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663585 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.663632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663599 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.663632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.663611 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.665189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.665120 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:46.665494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.665446 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:46.666445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.666415 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "kube-api-access-jfmfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:46.666555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.666511 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy" pod "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" (UID: "e3124d74-70b6-4a98-a039-d0a9a1c5c3f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:46.765359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.765272 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.765359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.765300 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.765359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.765308 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:46.765359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:46.765317 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmfp\" (UniqueName: \"kubernetes.io/projected/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4-kube-api-access-jfmfp\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:28:47.032351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.032247 4494 generic.go:356] "Generic (PLEG): container finished" podID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerID="45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508" exitCode=0 Nov 02 23:28:47.032351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.032316 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" Nov 02 23:28:47.032351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.032323 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerDied","Data":"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508"} Nov 02 23:28:47.032597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.032411 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2" event={"ID":"e3124d74-70b6-4a98-a039-d0a9a1c5c3f4","Type":"ContainerDied","Data":"a6d056c2916b6e98cfec3fe105b093669d252565ce33101b5e8e6e810c032f91"} Nov 02 23:28:47.032597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.032443 4494 scope.go:122] "RemoveContainer" containerID="45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508" Nov 02 23:28:47.043978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.043959 4494 scope.go:122] "RemoveContainer" containerID="b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486" Nov 02 23:28:47.052981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.052961 4494 scope.go:122] "RemoveContainer" containerID="45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508" Nov 02 23:28:47.053285 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:47.053265 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508\": container with ID starting with 45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508 not found: ID does not exist" containerID="45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508" Nov 02 23:28:47.053360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.053294 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508"} err="failed to get container status \"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508\": rpc error: code = NotFound desc = could not find container \"45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508\": container with ID starting with 45a3f5159f4520a246a8cec1d4c9d2a0abae4e39038d02f59af6872f367e2508 not found: ID does not exist" Nov 02 23:28:47.053360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.053313 4494 scope.go:122] "RemoveContainer" containerID="b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486" Nov 02 23:28:47.053563 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:47.053544 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486\": container with ID starting with b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486 not found: ID does not exist" containerID="b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486" Nov 02 23:28:47.053621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.053572 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486"} err="failed to get container status \"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486\": rpc error: code = NotFound desc = could not find container \"b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486\": container with ID starting with b04b89991d63ae15cbc1ee72166c2b7ba29004e3d6a497a807c3005b477c4486 not found: ID does not exist" Nov 02 23:28:47.200637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.200600 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:28:47.251582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.251548 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-03fe6e-openshift-default-7fb847b75c-jtcq2"] Nov 02 23:28:47.561055 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:28:47.561018 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:28:47.693668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:47.693631 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" path="/var/lib/kubelet/pods/e3124d74-70b6-4a98-a039-d0a9a1c5c3f4/volumes" Nov 02 23:28:50.480762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:50.480731 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:52.632218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:52.632155 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:52.793521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:52.793478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:53.418553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:53.418517 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:28:57.304840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:57.304806 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:57.456604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:57.456571 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:57.600817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:57.600732 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:28:58.455859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:28:58.455829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:01.674779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:01.674747 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:29:01.675297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:01.674967 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerName="maas-api" containerID="cri-o://b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7" gracePeriod=30 Nov 02 23:29:01.779381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:01.779339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:01.968922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:01.968895 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:29:02.086198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.082657 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\" (UniqueName: \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\") pod \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " Nov 02 23:29:02.086198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.082729 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") pod \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " Nov 02 23:29:02.086198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.082851 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\") pod \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " Nov 02 23:29:02.086198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.083110 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\") pod \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\" (UID: \"acdab9f9-9108-4064-a6bd-8a24587d3d8e\") " Nov 02 23:29:02.107927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.107883 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls" pod "acdab9f9-9108-4064-a6bd-8a24587d3d8e" (UID: "acdab9f9-9108-4064-a6bd-8a24587d3d8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:02.108219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.108134 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd" pod "acdab9f9-9108-4064-a6bd-8a24587d3d8e" (UID: "acdab9f9-9108-4064-a6bd-8a24587d3d8e"). InnerVolumeSpecName "kube-api-access-4dvnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:02.110232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.109327 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs" pod "acdab9f9-9108-4064-a6bd-8a24587d3d8e" (UID: "acdab9f9-9108-4064-a6bd-8a24587d3d8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:02.113439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.113410 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca" pod "acdab9f9-9108-4064-a6bd-8a24587d3d8e" (UID: "acdab9f9-9108-4064-a6bd-8a24587d3d8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:02.146506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.146476 4494 generic.go:356] "Generic (PLEG): container finished" podID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerID="b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7" exitCode=0 Nov 02 23:29:02.146690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.146524 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" Nov 02 23:29:02.146690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.146555 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" event={"ID":"acdab9f9-9108-4064-a6bd-8a24587d3d8e","Type":"ContainerDied","Data":"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7"} Nov 02 23:29:02.146690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.146616 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" event={"ID":"acdab9f9-9108-4064-a6bd-8a24587d3d8e","Type":"ContainerDied","Data":"7f6da53b244e0f839b83544d59f1cd5ac3388fe330d7309ad77ee643645cf50e"} Nov 02 23:29:02.146690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.146644 4494 scope.go:122] "RemoveContainer" containerID="b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7" Nov 02 23:29:02.158239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.158217 4494 scope.go:122] "RemoveContainer" containerID="b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7" Nov 02 23:29:02.158541 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:02.158521 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7\": container with ID starting with b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7 not found: ID does not exist" containerID="b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7" Nov 02 23:29:02.158602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.158551 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7"} err="failed to get container status \"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7\": rpc error: code = NotFound desc = could not find container \"b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7\": container with ID starting with b60f5559ba243c85983fd90578754dafbee8a00d4b9c6ef1c7dc273aaf0763c7 not found: ID does not exist" Nov 02 23:29:02.185475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.185410 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:02.185475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.185438 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdab9f9-9108-4064-a6bd-8a24587d3d8e-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:02.185475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.185449 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dvnd\" (UniqueName: \"kubernetes.io/projected/acdab9f9-9108-4064-a6bd-8a24587d3d8e-kube-api-access-4dvnd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:02.185475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.185460 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdab9f9-9108-4064-a6bd-8a24587d3d8e-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:02.313271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.313232 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:29:02.316860 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.316833 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr"] Nov 02 23:29:02.818994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.818958 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:29:02.819464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.819246 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" podUID="7be01df7-2de9-460b-a122-d6796ed35412" containerName="maas-api" containerID="cri-o://36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8" gracePeriod=30 Nov 02 23:29:02.921482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:02.921450 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:03.114406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.114383 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:29:03.155288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.155252 4494 generic.go:356] "Generic (PLEG): container finished" podID="7be01df7-2de9-460b-a122-d6796ed35412" containerID="36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8" exitCode=0 Nov 02 23:29:03.155451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.155297 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" Nov 02 23:29:03.155451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.155334 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" event={"ID":"7be01df7-2de9-460b-a122-d6796ed35412","Type":"ContainerDied","Data":"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8"} Nov 02 23:29:03.155451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.155387 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc" event={"ID":"7be01df7-2de9-460b-a122-d6796ed35412","Type":"ContainerDied","Data":"bb05aba55f6ccfce8b44c6ea924501360a7b59043a1ba9a7b0ce6ab9a3b6c437"} Nov 02 23:29:03.155451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.155411 4494 scope.go:122] "RemoveContainer" containerID="36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8" Nov 02 23:29:03.164688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.164669 4494 scope.go:122] "RemoveContainer" containerID="36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8" Nov 02 23:29:03.164940 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:03.164921 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8\": container with ID starting with 36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8 not found: ID does not exist" containerID="36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8" Nov 02 23:29:03.164990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.164952 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8"} err="failed to get container status \"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8\": rpc error: code = NotFound desc = could not find container \"36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8\": container with ID starting with 36f911200dc1c2df79433c0192aea03826683c850e305f557d6c4512327776b8 not found: ID does not exist" Nov 02 23:29:03.219198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.218641 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\" (UniqueName: \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\") pod \"7be01df7-2de9-460b-a122-d6796ed35412\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " Nov 02 23:29:03.219198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.218858 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") pod \"7be01df7-2de9-460b-a122-d6796ed35412\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " Nov 02 23:29:03.219198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.218982 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\") pod \"7be01df7-2de9-460b-a122-d6796ed35412\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " Nov 02 23:29:03.219198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.219016 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\") pod \"7be01df7-2de9-460b-a122-d6796ed35412\" (UID: \"7be01df7-2de9-460b-a122-d6796ed35412\") " Nov 02 23:29:03.224857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.221101 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca" pod "7be01df7-2de9-460b-a122-d6796ed35412" (UID: "7be01df7-2de9-460b-a122-d6796ed35412"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:03.229985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.229282 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls" pod "7be01df7-2de9-460b-a122-d6796ed35412" (UID: "7be01df7-2de9-460b-a122-d6796ed35412"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:03.229985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.229447 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb" pod "7be01df7-2de9-460b-a122-d6796ed35412" (UID: "7be01df7-2de9-460b-a122-d6796ed35412"). InnerVolumeSpecName "kube-api-access-jqvkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:03.231241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.230780 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs" pod "7be01df7-2de9-460b-a122-d6796ed35412" (UID: "7be01df7-2de9-460b-a122-d6796ed35412"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:03.320378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.320346 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:03.320378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.320380 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be01df7-2de9-460b-a122-d6796ed35412-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:03.320589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.320394 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be01df7-2de9-460b-a122-d6796ed35412-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:03.320589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.320404 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvkb\" (UniqueName: \"kubernetes.io/projected/7be01df7-2de9-460b-a122-d6796ed35412-kube-api-access-jqvkb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:03.577355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.577325 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:29:03.581528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.581496 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-11f7a8-56977cddb-mnhpc"] Nov 02 23:29:03.694512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.694478 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be01df7-2de9-460b-a122-d6796ed35412" path="/var/lib/kubelet/pods/7be01df7-2de9-460b-a122-d6796ed35412/volumes" Nov 02 23:29:03.694903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:03.694889 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" path="/var/lib/kubelet/pods/acdab9f9-9108-4064-a6bd-8a24587d3d8e/volumes" Nov 02 23:29:04.390341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:04.390309 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:04.534491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:04.534452 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:04.686341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:04.686304 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:04.916683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:04.916643 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a2f549-85c8987d88-l5skr" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.193:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:29:05.934040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:05.933878 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:06.405080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:06.405002 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:29:06.405268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:06.405236 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" containerID="cri-o://823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e" gracePeriod=30 Nov 02 23:29:06.427499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:06.427473 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:09.355667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:09.355609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:10.003131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:10.003095 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:11.683311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.683253 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:29:11.780248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.780213 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:11.826303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826274 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826319 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826373 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826409 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826454 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826500 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826553 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826598 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826649 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\") pod \"0f88f573-34a1-4832-857c-eff3178bbeb6\" (UID: \"0f88f573-34a1-4832-857c-eff3178bbeb6\") " Nov 02 23:29:11.826730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826720 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:11.827012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826772 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:11.827012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.826981 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:11.827332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.827128 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:11.827332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.827283 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:11.827332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.827288 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.827509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.827329 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f88f573-34a1-4832-857c-eff3178bbeb6-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.827509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.827350 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.828796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.828772 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:11.828796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.828780 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:11.828982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.828883 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:29:11.829436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.829411 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc" pod "0f88f573-34a1-4832-857c-eff3178bbeb6" (UID: "0f88f573-34a1-4832-857c-eff3178bbeb6"). InnerVolumeSpecName "kube-api-access-xm5tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:11.927237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.927197 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:11.928734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928707 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.928853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928736 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5tc\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-kube-api-access-xm5tc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.928853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928751 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.928853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928765 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.928853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928778 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:11.928853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:11.928790 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f88f573-34a1-4832-857c-eff3178bbeb6-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:12.221297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.221257 4494 generic.go:356] "Generic (PLEG): container finished" podID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerID="823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e" exitCode=0 Nov 02 23:29:12.221487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.221322 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" Nov 02 23:29:12.221487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.221349 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerDied","Data":"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e"} Nov 02 23:29:12.221487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.221428 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg" event={"ID":"0f88f573-34a1-4832-857c-eff3178bbeb6","Type":"ContainerDied","Data":"1a136effcfcbb009f15c071137d136b992a3271686d3ccbabcfa64a7c156aef3"} Nov 02 23:29:12.221487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.221450 4494 scope.go:122] "RemoveContainer" containerID="823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e" Nov 02 23:29:12.232541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.232524 4494 scope.go:122] "RemoveContainer" containerID="dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9" Nov 02 23:29:12.241260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.241240 4494 scope.go:122] "RemoveContainer" containerID="823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e" Nov 02 23:29:12.241539 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:12.241519 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e\": container with ID starting with 823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e not found: ID does not exist" containerID="823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e" Nov 02 23:29:12.241606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.241548 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e"} err="failed to get container status \"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e\": rpc error: code = NotFound desc = could not find container \"823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e\": container with ID starting with 823e10baada8d960ed00799b7bba910538b3a8bb768eea5edd7aeaffea3b613e not found: ID does not exist" Nov 02 23:29:12.241606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.241566 4494 scope.go:122] "RemoveContainer" containerID="dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9" Nov 02 23:29:12.241860 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:12.241843 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9\": container with ID starting with dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9 not found: ID does not exist" containerID="dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9" Nov 02 23:29:12.241924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.241865 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9"} err="failed to get container status \"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9\": rpc error: code = NotFound desc = could not find container \"dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9\": container with ID starting with dd2f1fe12fbb2597553e307a9558706bb134b01d9f7d8c72beae1e2abb849de9 not found: ID does not exist" Nov 02 23:29:12.388954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.388921 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:29:12.439034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:12.439004 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a2f549-openshift-default-7878dcbc49-cgpfg"] Nov 02 23:29:13.248066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:13.248029 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:13.696268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:13.696227 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" path="/var/lib/kubelet/pods/0f88f573-34a1-4832-857c-eff3178bbeb6/volumes" Nov 02 23:29:13.730953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:13.730920 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:29:13.731252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:13.731220 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" containerID="cri-o://056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009" gracePeriod=30 Nov 02 23:29:13.738653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:13.738628 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:14.490216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:14.490166 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:15.260311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:15.260254 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:17.562272 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:17.562189 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:29:18.990342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:18.990318 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:29:19.114586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114500 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114550 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114604 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114637 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114679 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114715 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.114820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114808 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.115089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114837 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:19.115089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114855 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.115089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114936 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:19.115089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.114962 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\") pod \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\" (UID: \"8f188c60-f265-4f8b-82da-726d3f2cd3b9\") " Nov 02 23:29:19.115337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115241 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:19.115337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115296 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:19.115563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115510 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:19.115563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115583 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.115818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115604 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.115818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115644 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.115818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.115654 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.117352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.117322 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:29:19.117475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.117362 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:19.117663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.117641 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:19.117776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.117682 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8" pod "8f188c60-f265-4f8b-82da-726d3f2cd3b9" (UID: "8f188c60-f265-4f8b-82da-726d3f2cd3b9"). InnerVolumeSpecName "kube-api-access-vnxb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:19.216982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.216946 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxb8\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-kube-api-access-vnxb8\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.216982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.216978 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.216982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.216988 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.217239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.216997 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.217239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.217005 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f188c60-f265-4f8b-82da-726d3f2cd3b9-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:19.264188 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.264137 4494 generic.go:356] "Generic (PLEG): container finished" podID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerID="056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009" exitCode=0 Nov 02 23:29:19.264374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.264209 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" Nov 02 23:29:19.264374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.264205 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerDied","Data":"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009"} Nov 02 23:29:19.264374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.264296 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw" event={"ID":"8f188c60-f265-4f8b-82da-726d3f2cd3b9","Type":"ContainerDied","Data":"34ce314f3dda1b0f4c24ec87a0570551f2bb9f8797dd0676ea89aeed6947a5d0"} Nov 02 23:29:19.264374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.264321 4494 scope.go:122] "RemoveContainer" containerID="056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009" Nov 02 23:29:19.275257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.275241 4494 scope.go:122] "RemoveContainer" containerID="402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29" Nov 02 23:29:19.283888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.283869 4494 scope.go:122] "RemoveContainer" containerID="056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009" Nov 02 23:29:19.284155 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:19.284138 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009\": container with ID starting with 056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009 not found: ID does not exist" containerID="056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009" Nov 02 23:29:19.284214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.284164 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009"} err="failed to get container status \"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009\": rpc error: code = NotFound desc = could not find container \"056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009\": container with ID starting with 056865214e81a85683a053cb106d98fd306eddad9d59d13a968f596f1dc06009 not found: ID does not exist" Nov 02 23:29:19.284214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.284203 4494 scope.go:122] "RemoveContainer" containerID="402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29" Nov 02 23:29:19.284466 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:19.284450 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29\": container with ID starting with 402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29 not found: ID does not exist" containerID="402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29" Nov 02 23:29:19.284508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.284474 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29"} err="failed to get container status \"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29\": rpc error: code = NotFound desc = could not find container \"402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29\": container with ID starting with 402e6631e483978483044bfd8d88d045fcf42e8a50a6792682aa31a113a51e29 not found: ID does not exist" Nov 02 23:29:19.431638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.431603 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:29:19.437167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.437140 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-11f7a8-openshift-default-5497f986bf-z2bjw"] Nov 02 23:29:19.694826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:19.694749 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" path="/var/lib/kubelet/pods/8f188c60-f265-4f8b-82da-726d3f2cd3b9/volumes" Nov 02 23:29:20.475233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:20.475199 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:20.619610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:20.619549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:21.024737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.024707 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w"] Nov 02 23:29:21.025326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025310 4494 cpu_manager.go:414] "containerMap: removing container" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerName="maas-api" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025327 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerName="maas-api" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025341 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025346 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025373 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025381 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025391 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.025398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025397 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025409 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025414 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025424 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025428 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025436 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7be01df7-2de9-460b-a122-d6796ed35412" containerName="maas-api" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025441 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be01df7-2de9-460b-a122-d6796ed35412" containerName="maas-api" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025450 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025454 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025538 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025545 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025551 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025556 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3124d74-70b6-4a98-a039-d0a9a1c5c3f4" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025562 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be01df7-2de9-460b-a122-d6796ed35412" containerName="maas-api" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025568 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f88f573-34a1-4832-857c-eff3178bbeb6" containerName="istio-proxy" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025577 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdab9f9-9108-4064-a6bd-8a24587d3d8e" containerName="maas-api" Nov 02 23:29:21.025636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.025583 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f188c60-f265-4f8b-82da-726d3f2cd3b9" containerName="istio-proxy" Nov 02 23:29:21.030394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.030376 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.086031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.085988 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w"] Nov 02 23:29:21.132869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.132827 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:29:21.132869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.132873 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:29:21.137884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.137855 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdhp8\" (UniqueName: \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.138011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.137911 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.138051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.138021 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.238967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.238932 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdhp8\" (UniqueName: \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.239148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.238983 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.239148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.239039 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.239706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.239688 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.241418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.241400 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.301378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.301305 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdhp8\" (UniqueName: \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\") pod \"maas-api-revoke-sub-e5eee018-4mp8w\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.343742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.343699 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:21.567570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.567477 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:21.711758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.711714 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:21.723732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:21.723710 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w"] Nov 02 23:29:21.724846 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:29:21.724822 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc44259_a2bb_43a7_a475_ad2c41611185.slice/crio-a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f WatchSource:0}: Error finding container a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f: Status 404 returned error can't find the container with id a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f Nov 02 23:29:22.284384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:22.284295 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" event={"ID":"3dc44259-a2bb-43a7-a475-ad2c41611185","Type":"ContainerStarted","Data":"8b7b200c7414198ed80d8ef63dbc3eee32ec5888c897badce0998b289e2ef460"} Nov 02 23:29:22.284384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:22.284353 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" event={"ID":"3dc44259-a2bb-43a7-a475-ad2c41611185","Type":"ContainerStarted","Data":"a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f"} Nov 02 23:29:22.403222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:22.403141 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" podStartSLOduration=2.403128689 podStartE2EDuration="2.403128689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:22.40064542 +0000 UTC m=+2476.830515282" watchObservedRunningTime="2026-11-02 23:29:22.403128689 +0000 UTC m=+2476.832998550" Nov 02 23:29:23.293273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:23.293236 4494 generic.go:356] "Generic (PLEG): container finished" podID="3dc44259-a2bb-43a7-a475-ad2c41611185" containerID="8b7b200c7414198ed80d8ef63dbc3eee32ec5888c897badce0998b289e2ef460" exitCode=0 Nov 02 23:29:23.293673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:23.293289 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" event={"ID":"3dc44259-a2bb-43a7-a475-ad2c41611185","Type":"ContainerDied","Data":"8b7b200c7414198ed80d8ef63dbc3eee32ec5888c897badce0998b289e2ef460"} Nov 02 23:29:24.460821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.460790 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:24.577592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.577500 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\") pod \"3dc44259-a2bb-43a7-a475-ad2c41611185\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " Nov 02 23:29:24.578030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.578002 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\" (UniqueName: \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\") pod \"3dc44259-a2bb-43a7-a475-ad2c41611185\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " Nov 02 23:29:24.578166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.578120 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\") pod \"3dc44259-a2bb-43a7-a475-ad2c41611185\" (UID: \"3dc44259-a2bb-43a7-a475-ad2c41611185\") " Nov 02 23:29:24.578709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.578586 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca" pod "3dc44259-a2bb-43a7-a475-ad2c41611185" (UID: "3dc44259-a2bb-43a7-a475-ad2c41611185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:24.578855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.578737 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.580718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.580690 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca" pod "3dc44259-a2bb-43a7-a475-ad2c41611185" (UID: "3dc44259-a2bb-43a7-a475-ad2c41611185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:24.581011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.580972 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8" pod "3dc44259-a2bb-43a7-a475-ad2c41611185" (UID: "3dc44259-a2bb-43a7-a475-ad2c41611185"). InnerVolumeSpecName "kube-api-access-pdhp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:24.679665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.679630 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdhp8\" (UniqueName: \"kubernetes.io/projected/3dc44259-a2bb-43a7-a475-ad2c41611185-kube-api-access-pdhp8\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.679665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:24.679659 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc44259-a2bb-43a7-a475-ad2c41611185-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:25.306654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:25.306624 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" Nov 02 23:29:25.306846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:25.306666 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w" event={"ID":"3dc44259-a2bb-43a7-a475-ad2c41611185","Type":"ContainerDied","Data":"a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f"} Nov 02 23:29:25.306846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:25.306720 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f7d4f114bdbe446fd5174db390e974a83615b82989b96adc509ebfd5ceac4f" Nov 02 23:29:25.420442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:25.420399 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:29:25.907959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:25.907907 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:26.539053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:26.539013 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:32.680531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:32.680491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:34.789364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:34.789318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:36.403942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.403908 4494 generic.go:356] "Generic (PLEG): container finished" podID="44e41bd5-3d46-4b8d-9623-709439954870" containerID="1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc" exitCode=0 Nov 02 23:29:36.404487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.403983 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerDied","Data":"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc"} Nov 02 23:29:36.404487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.404476 4494 scope.go:122] "RemoveContainer" containerID="1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc" Nov 02 23:29:36.406004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.405966 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:36.406132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.406061 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:36.406132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:36.406106 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:37.412079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:37.412045 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerStarted","Data":"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756"} Nov 02 23:29:37.530225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:37.530159 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:29:37.555803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:37.555770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:38.184565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.184528 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:29:38.185561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.185539 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 23:29:38.185639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.185598 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 23:29:38.972064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.972019 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j"] Nov 02 23:29:38.972867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.972844 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3dc44259-a2bb-43a7-a475-ad2c41611185" containerName="revoke-keys" Nov 02 23:29:38.972984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.972869 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc44259-a2bb-43a7-a475-ad2c41611185" containerName="revoke-keys" Nov 02 23:29:38.973053 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.973036 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc44259-a2bb-43a7-a475-ad2c41611185" containerName="revoke-keys" Nov 02 23:29:38.975387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:38.975365 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.046211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.038679 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j"] Nov 02 23:29:39.077506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.077472 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.077506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.077506 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.077800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.077636 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzzn\" (UniqueName: \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.080695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.080651 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:29:39.080849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.080758 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:29:39.178967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.178930 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.178967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.178967 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.179288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.179011 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzzn\" (UniqueName: \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.179755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.179727 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.181585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.181563 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.184657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.184637 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 23:29:39.184712 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.184689 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 23:29:39.257098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.257017 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzzn\" (UniqueName: \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\") pod \"maas-api-revoke-sub-80628867-dg62j\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.287836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.287787 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:39.683723 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:39.683694 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j"] Nov 02 23:29:39.685089 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:29:39.685065 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca28050_3f6b_4cab_a72f_c8916c07a8ce.slice/crio-8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b WatchSource:0}: Error finding container 8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b: Status 404 returned error can't find the container with id 8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b Nov 02 23:29:40.184965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:40.184931 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 23:29:40.185493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:40.184983 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 23:29:40.437199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:40.431231 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" event={"ID":"2ca28050-3f6b-4cab-a72f-c8916c07a8ce","Type":"ContainerStarted","Data":"a0eb5707de0967f256692937f59679c705a64057a38f9dd7563ade0420f1ddf3"} Nov 02 23:29:40.437199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:40.431298 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" event={"ID":"2ca28050-3f6b-4cab-a72f-c8916c07a8ce","Type":"ContainerStarted","Data":"8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b"} Nov 02 23:29:40.559433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:40.559337 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" podStartSLOduration=2.559316333 podStartE2EDuration="2.559316333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:40.557982786 +0000 UTC m=+2494.987852648" watchObservedRunningTime="2026-11-02 23:29:40.559316333 +0000 UTC m=+2494.989186200" Nov 02 23:29:41.184840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:41.184805 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 23:29:41.185026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:41.184858 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 23:29:41.460987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:41.460899 4494 generic.go:356] "Generic (PLEG): container finished" podID="2ca28050-3f6b-4cab-a72f-c8916c07a8ce" containerID="a0eb5707de0967f256692937f59679c705a64057a38f9dd7563ade0420f1ddf3" exitCode=0 Nov 02 23:29:41.461139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:41.460993 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" event={"ID":"2ca28050-3f6b-4cab-a72f-c8916c07a8ce","Type":"ContainerDied","Data":"a0eb5707de0967f256692937f59679c705a64057a38f9dd7563ade0420f1ddf3"} Nov 02 23:29:42.184818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.184787 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 23:29:42.185002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.184840 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 23:29:42.572598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.572559 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:42.622459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.622429 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:42.724735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.724649 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\") pod \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " Nov 02 23:29:42.724899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.724781 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\" (UniqueName: \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\") pod \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " Nov 02 23:29:42.724899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.724856 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\") pod \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\" (UID: \"2ca28050-3f6b-4cab-a72f-c8916c07a8ce\") " Nov 02 23:29:42.725038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.724997 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca" pod "2ca28050-3f6b-4cab-a72f-c8916c07a8ce" (UID: "2ca28050-3f6b-4cab-a72f-c8916c07a8ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:42.725260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.725242 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.726997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.726975 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca" pod "2ca28050-3f6b-4cab-a72f-c8916c07a8ce" (UID: "2ca28050-3f6b-4cab-a72f-c8916c07a8ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:29:42.727099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.727052 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn" pod "2ca28050-3f6b-4cab-a72f-c8916c07a8ce" (UID: "2ca28050-3f6b-4cab-a72f-c8916c07a8ce"). InnerVolumeSpecName "kube-api-access-lqzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:42.826778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.826748 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqzzn\" (UniqueName: \"kubernetes.io/projected/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-kube-api-access-lqzzn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:42.826778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:42.826774 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca28050-3f6b-4cab-a72f-c8916c07a8ce-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:43.188957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:43.188929 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:29:43.190155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:43.190133 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:29:43.476498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:43.476411 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" Nov 02 23:29:43.476498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:43.476433 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j" event={"ID":"2ca28050-3f6b-4cab-a72f-c8916c07a8ce","Type":"ContainerDied","Data":"8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b"} Nov 02 23:29:43.476498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:43.476482 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8865ab0f59a98b9213faa66a8ec81946df4a1cecba7fd077adad082049df8b4b" Nov 02 23:29:47.559725 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:29:47.559685 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:29:47.634034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:47.633996 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:49.207927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:49.207896 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:49.698065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:49.698035 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:49.841915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:49.841870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:52.404070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.404034 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:29:52.404870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.404841 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2ca28050-3f6b-4cab-a72f-c8916c07a8ce" containerName="revoke-keys" Nov 02 23:29:52.404870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.404870 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca28050-3f6b-4cab-a72f-c8916c07a8ce" containerName="revoke-keys" Nov 02 23:29:52.405133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.405063 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca28050-3f6b-4cab-a72f-c8916c07a8ce" containerName="revoke-keys" Nov 02 23:29:52.407569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.407546 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.418972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.418946 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:29:52.421449 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.421432 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.460983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.460958 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-818fe8-openshift-default-dockercfg-j2vfc\"" Nov 02 23:29:52.505819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.505786 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:29:52.548323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548285 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.548533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548341 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.548533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ng6f\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.548704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548565 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.548704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548601 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.548704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548682 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.548900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548728 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.548900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548766 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.548900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548860 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.549037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548953 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.549037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.548992 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.549037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549022 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.549208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549092 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.549208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549133 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncz99\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.549208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549197 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.549372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549214 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.549372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549266 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.549372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.549302 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.576060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.576031 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:29:52.650329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650299 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ng6f\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.650529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650362 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.650529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650392 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.650529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650420 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.650743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650562 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.650743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650622 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.650743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650661 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.650743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650715 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650749 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650774 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650807 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650838 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncz99\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650880 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650902 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650959 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.650962 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651024 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651071 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651118 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651188 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651281 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651418 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651552 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.651740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651647 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.652335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.651867 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.652335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.652093 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.652471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.652352 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.653518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.653493 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.654374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.654323 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.654465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.654431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.654542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.654526 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.712067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.712037 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.712249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.712141 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ng6f\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.712407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.712389 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.712759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.712739 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncz99\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\") pod \"e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.722421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.722396 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:52.733033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.733010 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:52.841649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:52.841605 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:53.157787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.157757 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:29:53.157868 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:29:53.157799 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe9f9cf_3e4c_4712_95f7_a2cf25e3804d.slice/crio-65c965fa0c437efa2a1de154d290e3a317082a82f33afc772ae31e0583dd8108 WatchSource:0}: Error finding container 65c965fa0c437efa2a1de154d290e3a317082a82f33afc772ae31e0583dd8108: Status 404 returned error can't find the container with id 65c965fa0c437efa2a1de154d290e3a317082a82f33afc772ae31e0583dd8108 Nov 02 23:29:53.159915 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:29:53.159886 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1113f649_222b_43fe_b68d_d3138cf68354.slice/crio-2e2da3cffdbfc4b5d6d70a4f37e061116ada345bbccff96d570b37cf125fa01e WatchSource:0}: Error finding container 2e2da3cffdbfc4b5d6d70a4f37e061116ada345bbccff96d570b37cf125fa01e: Status 404 returned error can't find the container with id 2e2da3cffdbfc4b5d6d70a4f37e061116ada345bbccff96d570b37cf125fa01e Nov 02 23:29:53.160568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.160474 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.160568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.160548 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.160699 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.160583 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.161394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.161371 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:29:53.162273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.162245 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.162378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.162300 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.162378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.162340 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:29:53.552902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.552870 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" event={"ID":"1113f649-222b-43fe-b68d-d3138cf68354","Type":"ContainerStarted","Data":"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937"} Nov 02 23:29:53.553554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.552924 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" event={"ID":"1113f649-222b-43fe-b68d-d3138cf68354","Type":"ContainerStarted","Data":"2e2da3cffdbfc4b5d6d70a4f37e061116ada345bbccff96d570b37cf125fa01e"} Nov 02 23:29:53.555973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.555949 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" event={"ID":"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d","Type":"ContainerStarted","Data":"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7"} Nov 02 23:29:53.555973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.555988 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" event={"ID":"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d","Type":"ContainerStarted","Data":"65c965fa0c437efa2a1de154d290e3a317082a82f33afc772ae31e0583dd8108"} Nov 02 23:29:53.681030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.680944 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podStartSLOduration=1.680919399 podStartE2EDuration="1.680919399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:53.678692859 +0000 UTC m=+2508.108562760" watchObservedRunningTime="2026-11-02 23:29:53.680919399 +0000 UTC m=+2508.110789268" Nov 02 23:29:53.723435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.723401 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:53.762203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.749320 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:53.824522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:53.824349 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podStartSLOduration=1.8243301 podStartE2EDuration="1.8243301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:53.820412779 +0000 UTC m=+2508.250282642" watchObservedRunningTime="2026-11-02 23:29:53.8243301 +0000 UTC m=+2508.254199962" Nov 02 23:29:54.724117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:54.724081 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 23:29:54.724598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:54.724144 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 23:29:54.742445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:54.742302 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 23:29:54.742445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:54.742372 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 23:29:55.722999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:55.722969 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 23:29:55.723209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:55.723016 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 23:29:55.749207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:55.743577 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 23:29:55.749207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:55.743637 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 23:29:56.279102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.279062 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b"] Nov 02 23:29:56.285680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.285087 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.321844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.321810 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b"] Nov 02 23:29:56.373202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.369059 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:29:56.373202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.369871 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:29:56.418905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.418871 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjww\" (UniqueName: \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.419108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.419026 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.419108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.419071 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.520052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.520012 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjww\" (UniqueName: \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.520274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.520119 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.520274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.520158 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.520884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.520859 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.523005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.522983 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.589272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.588627 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjww\" (UniqueName: \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\") pod \"maas-api-revoke-sub-b6e6fa27-q275b\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.598397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.598367 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:56.723007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.722970 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 23:29:56.723191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.723019 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 23:29:56.754198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.740589 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 23:29:56.754198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:56.740661 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 23:29:57.021988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.021963 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b"] Nov 02 23:29:57.025783 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:29:57.024837 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d7a4ee_6f3f_4cee_bbcf_87b831bdef8c.slice/crio-eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941 WatchSource:0}: Error finding container eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941: Status 404 returned error can't find the container with id eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941 Nov 02 23:29:57.608425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.608349 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" event={"ID":"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c","Type":"ContainerStarted","Data":"e40ba40f90b494780d83078f3c6d57312378c0bb7b8485cc993ca64de3376b23"} Nov 02 23:29:57.608425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.608397 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" event={"ID":"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c","Type":"ContainerStarted","Data":"eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941"} Nov 02 23:29:57.723915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.723884 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 23:29:57.724105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.723929 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 23:29:57.727900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.727846 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" podStartSLOduration=1.727831449 podStartE2EDuration="1.727831449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:57.727588047 +0000 UTC m=+2512.157457908" watchObservedRunningTime="2026-11-02 23:29:57.727831449 +0000 UTC m=+2512.157701315" Nov 02 23:29:57.759198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.750275 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 23:29:57.759198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:57.750338 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 23:29:58.282982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.282749 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:29:58.616530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.616447 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" event={"ID":"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c","Type":"ContainerDied","Data":"e40ba40f90b494780d83078f3c6d57312378c0bb7b8485cc993ca64de3376b23"} Nov 02 23:29:58.617517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.616354 4494 generic.go:356] "Generic (PLEG): container finished" podID="02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" containerID="e40ba40f90b494780d83078f3c6d57312378c0bb7b8485cc993ca64de3376b23" exitCode=0 Nov 02 23:29:58.723211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.723164 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 02 23:29:58.723377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.723229 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 02 23:29:58.758197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.750687 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 02 23:29:58.758197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:58.750740 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 02 23:29:59.727063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.727022 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:59.727608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.727489 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:59.728142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.728119 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:29:59.737409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.737387 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:59.737677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.737659 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:59.738361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.738344 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:29:59.787686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.787664 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:29:59.905640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.905558 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\" (UniqueName: \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\") pod \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " Nov 02 23:29:59.905640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.905626 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\") pod \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " Nov 02 23:29:59.905835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.905659 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\") pod \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\" (UID: \"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c\") " Nov 02 23:29:59.906040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.906015 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca" pod "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" (UID: "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:59.906462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.906439 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:29:59.908266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.908246 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww" pod "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" (UID: "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c"). InnerVolumeSpecName "kube-api-access-dnjww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:59.908350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:29:59.908288 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca" pod "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" (UID: "02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:00.007366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.007327 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnjww\" (UniqueName: \"kubernetes.io/projected/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-kube-api-access-dnjww\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:00.007366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.007361 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:00.180780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.180739 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:00.182017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.181984 4494 cpu_manager.go:414] "containerMap: removing container" podUID="02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" containerName="revoke-keys" Nov 02 23:30:00.182145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.182018 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" containerName="revoke-keys" Nov 02 23:30:00.182237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.182222 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" containerName="revoke-keys" Nov 02 23:30:00.185438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.185401 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:00.185576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.185551 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.189187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.189147 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.194004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.193982 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:00.197030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.197010 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.201428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.201401 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:00.204697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.204677 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.218758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.218735 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:00.221708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.221690 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.237482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.237278 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:00.266785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.264384 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:00.279487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.279456 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:00.285220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.284258 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:00.293905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.293865 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:00.310876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.310845 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.311038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.310890 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgph\" (UniqueName: \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.311096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311073 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.311154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311132 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.311280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311260 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.311329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311313 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kh6\" (UniqueName: \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.311367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311344 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5l8q\" (UniqueName: \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.311459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311439 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfmg\" (UniqueName: \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.311546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311530 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwcx\" (UniqueName: \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.311613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.311593 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.412769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.412736 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.412976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.412781 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kh6\" (UniqueName: \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.412976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.412804 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5l8q\" (UniqueName: \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.412976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.412912 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfmg\" (UniqueName: \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.413122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413004 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwcx\" (UniqueName: \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.413122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413041 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.413241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413140 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.413241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413199 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgph\" (UniqueName: \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.413348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.413405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413372 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.413582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413555 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.413738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413719 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.413956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413941 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.413995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.413939 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.414064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.414049 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.475669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.475573 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwcx\" (UniqueName: \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\") pod \"maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.475669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.475652 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfmg\" (UniqueName: \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\") pod \"maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.475921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.475787 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kh6\" (UniqueName: \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.475921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.475784 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5l8q\" (UniqueName: \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.475921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.475859 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgph\" (UniqueName: \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\") pod \"maas-api-key-cleanup-29894370-pjgk5\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.504222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.504189 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:00.513055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.513036 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:00.521981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.521951 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:00.528161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.528135 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:00.548367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.544695 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:00.645019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.644994 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" Nov 02 23:30:00.646113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.645245 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b" event={"ID":"02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c","Type":"ContainerDied","Data":"eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941"} Nov 02 23:30:00.646113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.645299 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb84e9a16d9488fee1057093e6c55de88af2ff4f4753d14e312da5f57cd85941" Nov 02 23:30:00.897021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.896997 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:00.898032 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:00.898004 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6962aba_844b_4335_a495_4345f04c687f.slice/crio-453d153b02f6eafe6b2eba410475b215df9998912d200d5ce2db91722aca9c67 WatchSource:0}: Error finding container 453d153b02f6eafe6b2eba410475b215df9998912d200d5ce2db91722aca9c67: Status 404 returned error can't find the container with id 453d153b02f6eafe6b2eba410475b215df9998912d200d5ce2db91722aca9c67 Nov 02 23:30:00.948426 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:00.948397 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83a3a4e_755f_4a3e_8876_473fc6eccb1b.slice/crio-e2a92842311c4b4bec747e0395bf59cd66ca6c929e0bcd03ee4ab66acb73842f WatchSource:0}: Error finding container e2a92842311c4b4bec747e0395bf59cd66ca6c929e0bcd03ee4ab66acb73842f: Status 404 returned error can't find the container with id e2a92842311c4b4bec747e0395bf59cd66ca6c929e0bcd03ee4ab66acb73842f Nov 02 23:30:00.948997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.948980 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:00.951214 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:00.951150 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23b00f3_bfd5_4532_9c96_d77ee5af7926.slice/crio-af321d690c1d51038f5df59d8981f7bf36cc0b53e35c7eb06c91486eb3524fdf WatchSource:0}: Error finding container af321d690c1d51038f5df59d8981f7bf36cc0b53e35c7eb06c91486eb3524fdf: Status 404 returned error can't find the container with id af321d690c1d51038f5df59d8981f7bf36cc0b53e35c7eb06c91486eb3524fdf Nov 02 23:30:00.952667 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:00.952641 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185bd76d_5c8e_4dc8_b942_e1fb0fb94a3b.slice/crio-b2c344a59bc1ca7540b68747ce7cff7af27c088a8fc2b738f31a7c41eb01db4e WatchSource:0}: Error finding container b2c344a59bc1ca7540b68747ce7cff7af27c088a8fc2b738f31a7c41eb01db4e: Status 404 returned error can't find the container with id b2c344a59bc1ca7540b68747ce7cff7af27c088a8fc2b738f31a7c41eb01db4e Nov 02 23:30:00.953268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.953240 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:00.956431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:00.956407 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:01.017776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.017708 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:01.652629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.652594 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerStarted","Data":"ba20cb21b8cc62309369cd24c3bad111f68d2707cea5c21b8c35c0d4d1e25348"} Nov 02 23:30:01.652832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.652661 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerStarted","Data":"e2a92842311c4b4bec747e0395bf59cd66ca6c929e0bcd03ee4ab66acb73842f"} Nov 02 23:30:01.654399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.654366 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerStarted","Data":"8db3ab20c21deb2d7760d3c47b74d0fc516dbd9ad81b2a754a1a1c5bcba79eb5"} Nov 02 23:30:01.654525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.654420 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerStarted","Data":"60726306223ce545e5614a9034023fef7cf6be248c4fbac016a766a7cca4b518"} Nov 02 23:30:01.656225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.656199 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerStarted","Data":"8667bf8e6c7760806d2a0d520e6026c8496cabd1040060d6edc932d546d90ba2"} Nov 02 23:30:01.656365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.656243 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerStarted","Data":"af321d690c1d51038f5df59d8981f7bf36cc0b53e35c7eb06c91486eb3524fdf"} Nov 02 23:30:01.657857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.657828 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerStarted","Data":"f79b1ace9db3c8683d5f8b444d28274854ab6993b0c654510475a0b0a7bead0c"} Nov 02 23:30:01.657988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.657873 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerStarted","Data":"b2c344a59bc1ca7540b68747ce7cff7af27c088a8fc2b738f31a7c41eb01db4e"} Nov 02 23:30:01.659769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.659741 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerStarted","Data":"895c1d732f6ea72d3f6929d26166c5a247b1d0845d68eef8647099302dc9d4e0"} Nov 02 23:30:01.659880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.659790 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerStarted","Data":"453d153b02f6eafe6b2eba410475b215df9998912d200d5ce2db91722aca9c67"} Nov 02 23:30:01.770863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:01.770807 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" podStartSLOduration=1.770794465 podStartE2EDuration="1.770794465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:01.770389477 +0000 UTC m=+2516.200259338" watchObservedRunningTime="2026-11-02 23:30:01.770794465 +0000 UTC m=+2516.200664325" Nov 02 23:30:02.004275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.004218 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" podStartSLOduration=2.004200604 podStartE2EDuration="2.004200604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:01.887081107 +0000 UTC m=+2516.316950968" watchObservedRunningTime="2026-11-02 23:30:02.004200604 +0000 UTC m=+2516.434070468" Nov 02 23:30:02.005945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.005908 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" podStartSLOduration=2.005890785 podStartE2EDuration="2.005890785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.003030593 +0000 UTC m=+2516.432900455" watchObservedRunningTime="2026-11-02 23:30:02.005890785 +0000 UTC m=+2516.435760645" Nov 02 23:30:02.123877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.123825 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" podStartSLOduration=2.123811328 podStartE2EDuration="2.123811328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.120731881 +0000 UTC m=+2516.550601741" watchObservedRunningTime="2026-11-02 23:30:02.123811328 +0000 UTC m=+2516.553681188" Nov 02 23:30:02.238028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.237913 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" podStartSLOduration=2.237890817 podStartE2EDuration="2.237890817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.237594461 +0000 UTC m=+2516.667464334" watchObservedRunningTime="2026-11-02 23:30:02.237890817 +0000 UTC m=+2516.667760677" Nov 02 23:30:02.668086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.668047 4494 generic.go:356] "Generic (PLEG): container finished" podID="c6962aba-844b-4335-a495-4345f04c687f" containerID="895c1d732f6ea72d3f6929d26166c5a247b1d0845d68eef8647099302dc9d4e0" exitCode=60 Nov 02 23:30:02.668281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.668138 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerDied","Data":"895c1d732f6ea72d3f6929d26166c5a247b1d0845d68eef8647099302dc9d4e0"} Nov 02 23:30:02.668497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.668462 4494 scope.go:122] "RemoveContainer" containerID="895c1d732f6ea72d3f6929d26166c5a247b1d0845d68eef8647099302dc9d4e0" Nov 02 23:30:02.669970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.669946 4494 generic.go:356] "Generic (PLEG): container finished" podID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerID="ba20cb21b8cc62309369cd24c3bad111f68d2707cea5c21b8c35c0d4d1e25348" exitCode=60 Nov 02 23:30:02.670067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.670033 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerDied","Data":"ba20cb21b8cc62309369cd24c3bad111f68d2707cea5c21b8c35c0d4d1e25348"} Nov 02 23:30:02.670244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.670230 4494 scope.go:122] "RemoveContainer" containerID="ba20cb21b8cc62309369cd24c3bad111f68d2707cea5c21b8c35c0d4d1e25348" Nov 02 23:30:02.671987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.671951 4494 generic.go:356] "Generic (PLEG): container finished" podID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerID="8db3ab20c21deb2d7760d3c47b74d0fc516dbd9ad81b2a754a1a1c5bcba79eb5" exitCode=60 Nov 02 23:30:02.672096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.671986 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerDied","Data":"8db3ab20c21deb2d7760d3c47b74d0fc516dbd9ad81b2a754a1a1c5bcba79eb5"} Nov 02 23:30:02.672245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.672223 4494 scope.go:122] "RemoveContainer" containerID="8db3ab20c21deb2d7760d3c47b74d0fc516dbd9ad81b2a754a1a1c5bcba79eb5" Nov 02 23:30:02.673901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.673867 4494 generic.go:356] "Generic (PLEG): container finished" podID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerID="8667bf8e6c7760806d2a0d520e6026c8496cabd1040060d6edc932d546d90ba2" exitCode=60 Nov 02 23:30:02.674027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.673928 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerDied","Data":"8667bf8e6c7760806d2a0d520e6026c8496cabd1040060d6edc932d546d90ba2"} Nov 02 23:30:02.674347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.674189 4494 scope.go:122] "RemoveContainer" containerID="8667bf8e6c7760806d2a0d520e6026c8496cabd1040060d6edc932d546d90ba2" Nov 02 23:30:02.675665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.675643 4494 generic.go:356] "Generic (PLEG): container finished" podID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerID="f79b1ace9db3c8683d5f8b444d28274854ab6993b0c654510475a0b0a7bead0c" exitCode=60 Nov 02 23:30:02.675749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.675720 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerDied","Data":"f79b1ace9db3c8683d5f8b444d28274854ab6993b0c654510475a0b0a7bead0c"} Nov 02 23:30:02.675854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:02.675842 4494 scope.go:122] "RemoveContainer" containerID="f79b1ace9db3c8683d5f8b444d28274854ab6993b0c654510475a0b0a7bead0c" Nov 02 23:30:03.694128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.694081 4494 generic.go:356] "Generic (PLEG): container finished" podID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerID="b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c" exitCode=60 Nov 02 23:30:03.694710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.694458 4494 scope.go:122] "RemoveContainer" containerID="b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c" Nov 02 23:30:03.694794 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:03.694740 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9_odh-ai-gateway-infra(75a3dc0a-6807-41aa-a60a-cb96cc9a7df9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" Nov 02 23:30:03.695216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.695187 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerStarted","Data":"4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701"} Nov 02 23:30:03.695345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.695242 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerStarted","Data":"c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c"} Nov 02 23:30:03.695345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.695264 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerDied","Data":"b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c"} Nov 02 23:30:03.695345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.695292 4494 scope.go:122] "RemoveContainer" containerID="8db3ab20c21deb2d7760d3c47b74d0fc516dbd9ad81b2a754a1a1c5bcba79eb5" Nov 02 23:30:03.696532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.696448 4494 generic.go:356] "Generic (PLEG): container finished" podID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerID="9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d" exitCode=60 Nov 02 23:30:03.696599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.696527 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerDied","Data":"9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d"} Nov 02 23:30:03.696738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.696722 4494 scope.go:122] "RemoveContainer" containerID="9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d" Nov 02 23:30:03.696983 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:03.696964 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh_odh-ai-gateway-infra(f23b00f3-bfd5-4532-9c96-d77ee5af7926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" Nov 02 23:30:03.698360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.698340 4494 generic.go:356] "Generic (PLEG): container finished" podID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerID="90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc" exitCode=60 Nov 02 23:30:03.698437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.698375 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerDied","Data":"90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc"} Nov 02 23:30:03.698568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.698555 4494 scope.go:122] "RemoveContainer" containerID="90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc" Nov 02 23:30:03.698763 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:03.698747 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-pjgk5_odh-ai-gateway-infra(185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" Nov 02 23:30:03.884260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.884237 4494 scope.go:122] "RemoveContainer" containerID="8667bf8e6c7760806d2a0d520e6026c8496cabd1040060d6edc932d546d90ba2" Nov 02 23:30:03.897920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:03.897892 4494 scope.go:122] "RemoveContainer" containerID="f79b1ace9db3c8683d5f8b444d28274854ab6993b0c654510475a0b0a7bead0c" Nov 02 23:30:04.707144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.707113 4494 generic.go:356] "Generic (PLEG): container finished" podID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerID="c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c" exitCode=60 Nov 02 23:30:04.707648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.707201 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerDied","Data":"c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c"} Nov 02 23:30:04.707648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.707258 4494 scope.go:122] "RemoveContainer" containerID="ba20cb21b8cc62309369cd24c3bad111f68d2707cea5c21b8c35c0d4d1e25348" Nov 02 23:30:04.707648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.707481 4494 scope.go:122] "RemoveContainer" containerID="c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c" Nov 02 23:30:04.707906 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:04.707879 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf_odh-ai-gateway-infra(b83a3a4e-755f-4a3e-8876-473fc6eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" Nov 02 23:30:04.709439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.709420 4494 scope.go:122] "RemoveContainer" containerID="b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c" Nov 02 23:30:04.709700 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:04.709677 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9_odh-ai-gateway-infra(75a3dc0a-6807-41aa-a60a-cb96cc9a7df9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" Nov 02 23:30:04.710997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.710975 4494 scope.go:122] "RemoveContainer" containerID="9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d" Nov 02 23:30:04.711212 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:04.711187 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh_odh-ai-gateway-infra(f23b00f3-bfd5-4532-9c96-d77ee5af7926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" Nov 02 23:30:04.712582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.712564 4494 scope.go:122] "RemoveContainer" containerID="90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc" Nov 02 23:30:04.712775 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:04.712754 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-pjgk5_odh-ai-gateway-infra(185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" Nov 02 23:30:04.714312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.714292 4494 generic.go:356] "Generic (PLEG): container finished" podID="c6962aba-844b-4335-a495-4345f04c687f" containerID="4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701" exitCode=60 Nov 02 23:30:04.714402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.714380 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerDied","Data":"4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701"} Nov 02 23:30:04.714528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.714517 4494 scope.go:122] "RemoveContainer" containerID="4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701" Nov 02 23:30:04.714698 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:04.714681 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h_odh-ai-gateway-infra(c6962aba-844b-4335-a495-4345f04c687f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" podUID="c6962aba-844b-4335-a495-4345f04c687f" Nov 02 23:30:04.722286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.722265 4494 scope.go:122] "RemoveContainer" containerID="895c1d732f6ea72d3f6929d26166c5a247b1d0845d68eef8647099302dc9d4e0" Nov 02 23:30:04.941601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:04.941562 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:05.722725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:05.722683 4494 scope.go:122] "RemoveContainer" containerID="c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c" Nov 02 23:30:05.723211 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:05.723002 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf_odh-ai-gateway-infra(b83a3a4e-755f-4a3e-8876-473fc6eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" Nov 02 23:30:05.724426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:05.724409 4494 scope.go:122] "RemoveContainer" containerID="4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701" Nov 02 23:30:05.724656 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:05.724639 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h_odh-ai-gateway-infra(c6962aba-844b-4335-a495-4345f04c687f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" podUID="c6962aba-844b-4335-a495-4345f04c687f" Nov 02 23:30:09.712203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:09.712157 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:09.878528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:09.878478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:11.220824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.220788 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:30:11.226524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.226503 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256416 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256462 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256487 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256575 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256614 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256676 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256728 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256759 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.256865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.256852 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxhv\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.279631 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.279605 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-08c2c6-openshift-default-dockercfg-qcc5b\"" Nov 02 23:30:11.334369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.334338 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:30:11.357725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357690 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.357906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357740 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.357906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357775 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.357906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357822 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357906 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.357985 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358048 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358102 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxhv\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358299 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358339 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358769 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.358949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358926 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.359037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.358978 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.359103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.359034 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.360770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.360744 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.361316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.361291 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.419142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.419117 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.419326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.419308 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxhv\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.541213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.541114 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:11.978590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.978564 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:30:11.979049 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:11.979023 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456a86bd_4db5_43f7_aea4_88e906b845fe.slice/crio-93c43c9305cd707f87fa70a51060c981855ebb2f3989fb92644d5b45358a9fdb WatchSource:0}: Error finding container 93c43c9305cd707f87fa70a51060c981855ebb2f3989fb92644d5b45358a9fdb: Status 404 returned error can't find the container with id 93c43c9305cd707f87fa70a51060c981855ebb2f3989fb92644d5b45358a9fdb Nov 02 23:30:11.981747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.981716 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:11.981816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.981782 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:11.981816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:11.981811 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:12.769812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:12.769767 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" event={"ID":"456a86bd-4db5-43f7-aea4-88e906b845fe","Type":"ContainerStarted","Data":"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f"} Nov 02 23:30:12.770257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:12.769827 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" event={"ID":"456a86bd-4db5-43f7-aea4-88e906b845fe","Type":"ContainerStarted","Data":"93c43c9305cd707f87fa70a51060c981855ebb2f3989fb92644d5b45358a9fdb"} Nov 02 23:30:12.779017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:12.778989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:12.975225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:12.975143 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podStartSLOduration=1.9751307009999999 podStartE2EDuration="1.975130701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:12.972300839 +0000 UTC m=+2527.402170709" watchObservedRunningTime="2026-11-02 23:30:12.975130701 +0000 UTC m=+2527.405000631" Nov 02 23:30:13.026844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:13.026762 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:30:13.546194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:13.542489 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:13.556461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:13.556420 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:13.556737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:13.556685 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:14.545198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:14.544268 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:14.545198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:14.544331 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:14.687038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:14.687004 4494 scope.go:122] "RemoveContainer" containerID="90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc" Nov 02 23:30:14.795532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:14.795449 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:15.200892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:15.200853 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:15.559258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:15.557470 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:15.559258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:15.557533 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:15.694623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:15.694591 4494 scope.go:122] "RemoveContainer" containerID="9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d" Nov 02 23:30:15.792892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:15.792861 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerStarted","Data":"b2a28354502432e5151a103f197974226714dda59e4f72b74e3fe17388158307"} Nov 02 23:30:16.541879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.541787 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:16.542189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.542139 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:16.686889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.686856 4494 scope.go:122] "RemoveContainer" containerID="b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c" Nov 02 23:30:16.687336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.687315 4494 scope.go:122] "RemoveContainer" containerID="4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701" Nov 02 23:30:16.801081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.800996 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerStarted","Data":"1bb67e26b9f68e0f65fb98aca525f88cbcc8bd0151bc9d5e68e086652d305966"} Nov 02 23:30:16.803029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.803003 4494 generic.go:356] "Generic (PLEG): container finished" podID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerID="b2a28354502432e5151a103f197974226714dda59e4f72b74e3fe17388158307" exitCode=60 Nov 02 23:30:16.803197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.803032 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerDied","Data":"b2a28354502432e5151a103f197974226714dda59e4f72b74e3fe17388158307"} Nov 02 23:30:16.803197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.803077 4494 scope.go:122] "RemoveContainer" containerID="90573eed782aeba36016e9fcc96456822f6b36b04746a8bcef49dbfe6ad985cc" Nov 02 23:30:16.803504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.803478 4494 scope.go:122] "RemoveContainer" containerID="b2a28354502432e5151a103f197974226714dda59e4f72b74e3fe17388158307" Nov 02 23:30:16.803804 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:16.803781 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-pjgk5_odh-ai-gateway-infra(185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" Nov 02 23:30:16.926283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:16.926253 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:17.543470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.543372 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:17.543768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.543720 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:17.599204 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:17.596350 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:30:17.700015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.699981 4494 scope.go:122] "RemoveContainer" containerID="c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c" Nov 02 23:30:17.814633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.814550 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerStarted","Data":"eff680950943b1323feeb74f8542ba3c4db9e78b9fad0b26f8eb61cddbeff768"} Nov 02 23:30:17.816861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.816835 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerStarted","Data":"ad21ab2513bbbc6913b037edf33ca59517a17989a3cbef30b83f04bfaaeb2b69"} Nov 02 23:30:17.818760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.818739 4494 generic.go:356] "Generic (PLEG): container finished" podID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerID="1bb67e26b9f68e0f65fb98aca525f88cbcc8bd0151bc9d5e68e086652d305966" exitCode=60 Nov 02 23:30:17.818963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.818807 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerDied","Data":"1bb67e26b9f68e0f65fb98aca525f88cbcc8bd0151bc9d5e68e086652d305966"} Nov 02 23:30:17.819105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.819095 4494 scope.go:122] "RemoveContainer" containerID="9931cf8fd26a36acacb75973ff2c54beac07cb988c03a3f1b96e168fde27819d" Nov 02 23:30:17.819628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.819583 4494 scope.go:122] "RemoveContainer" containerID="1bb67e26b9f68e0f65fb98aca525f88cbcc8bd0151bc9d5e68e086652d305966" Nov 02 23:30:17.820139 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:17.820086 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh_odh-ai-gateway-infra(f23b00f3-bfd5-4532-9c96-d77ee5af7926)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" Nov 02 23:30:17.955200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:17.930911 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:18.064499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.064473 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:18.171997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.171955 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\" (UniqueName: \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\") pod \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " Nov 02 23:30:18.171997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.171996 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\") pod \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\" (UID: \"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b\") " Nov 02 23:30:18.172474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.172446 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca" pod "185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" (UID: "185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:18.174226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.174199 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph" pod "185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" (UID: "185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b"). InnerVolumeSpecName "kube-api-access-4bgph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:18.273245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.273206 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgph\" (UniqueName: \"kubernetes.io/projected/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-kube-api-access-4bgph\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:18.273245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.273233 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:18.542161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.542127 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 23:30:18.542404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.542197 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 23:30:18.830681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.830550 4494 generic.go:356] "Generic (PLEG): container finished" podID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerID="a9ab7c9325025bfcc79d25d1471e8791cea25d15407ec281c9fe0a5d62f4039f" exitCode=60 Nov 02 23:30:18.830681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.830623 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerDied","Data":"a9ab7c9325025bfcc79d25d1471e8791cea25d15407ec281c9fe0a5d62f4039f"} Nov 02 23:30:18.831308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.830692 4494 scope.go:122] "RemoveContainer" containerID="c6f92f0a761a6aec04df5555fcfb9e2e047f4d519a9ef7c2cb4dd8a38db4855c" Nov 02 23:30:18.831308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.831134 4494 scope.go:122] "RemoveContainer" containerID="a9ab7c9325025bfcc79d25d1471e8791cea25d15407ec281c9fe0a5d62f4039f" Nov 02 23:30:18.831536 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:18.831519 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf_odh-ai-gateway-infra(b83a3a4e-755f-4a3e-8876-473fc6eccb1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" Nov 02 23:30:18.832835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.832812 4494 generic.go:356] "Generic (PLEG): container finished" podID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerID="ad21ab2513bbbc6913b037edf33ca59517a17989a3cbef30b83f04bfaaeb2b69" exitCode=60 Nov 02 23:30:18.833000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.832880 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerDied","Data":"ad21ab2513bbbc6913b037edf33ca59517a17989a3cbef30b83f04bfaaeb2b69"} Nov 02 23:30:18.833310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.833287 4494 scope.go:122] "RemoveContainer" containerID="ad21ab2513bbbc6913b037edf33ca59517a17989a3cbef30b83f04bfaaeb2b69" Nov 02 23:30:18.833691 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:18.833659 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9_odh-ai-gateway-infra(75a3dc0a-6807-41aa-a60a-cb96cc9a7df9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" Nov 02 23:30:18.836350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.836328 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" Nov 02 23:30:18.836462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.836364 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5" event={"ID":"185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b","Type":"ContainerDied","Data":"b2c344a59bc1ca7540b68747ce7cff7af27c088a8fc2b738f31a7c41eb01db4e"} Nov 02 23:30:18.838797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.838772 4494 generic.go:356] "Generic (PLEG): container finished" podID="c6962aba-844b-4335-a495-4345f04c687f" containerID="eff680950943b1323feeb74f8542ba3c4db9e78b9fad0b26f8eb61cddbeff768" exitCode=60 Nov 02 23:30:18.838922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.838846 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerDied","Data":"eff680950943b1323feeb74f8542ba3c4db9e78b9fad0b26f8eb61cddbeff768"} Nov 02 23:30:18.839278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.839249 4494 scope.go:122] "RemoveContainer" containerID="eff680950943b1323feeb74f8542ba3c4db9e78b9fad0b26f8eb61cddbeff768" Nov 02 23:30:18.839564 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:18.839543 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h_odh-ai-gateway-infra(c6962aba-844b-4335-a495-4345f04c687f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" podUID="c6962aba-844b-4335-a495-4345f04c687f" Nov 02 23:30:18.845028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.844985 4494 scope.go:122] "RemoveContainer" containerID="b399eb0e88976a52ab648da58805dab7e2d509ccbe49121d0eeeb9d8166a1b8c" Nov 02 23:30:18.859514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.859488 4494 scope.go:122] "RemoveContainer" containerID="b2a28354502432e5151a103f197974226714dda59e4f72b74e3fe17388158307" Nov 02 23:30:18.870627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.870603 4494 scope.go:122] "RemoveContainer" containerID="4ccec2a7aaa497d8c307a96d5d88bef25e2500581c3d9f502e1c0081206c9701" Nov 02 23:30:18.953429 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:18.953393 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:19.012895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.012870 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:19.061012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.060981 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:19.068755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.068728 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-pjgk5"] Nov 02 23:30:19.082970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.082873 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\" (UniqueName: \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\") pod \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " Nov 02 23:30:19.082970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.082941 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\") pod \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\" (UID: \"f23b00f3-bfd5-4532-9c96-d77ee5af7926\") " Nov 02 23:30:19.083359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.083325 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca" pod "f23b00f3-bfd5-4532-9c96-d77ee5af7926" (UID: "f23b00f3-bfd5-4532-9c96-d77ee5af7926"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.083537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.083520 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23b00f3-bfd5-4532-9c96-d77ee5af7926-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.085273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.085245 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q" pod "f23b00f3-bfd5-4532-9c96-d77ee5af7926" (UID: "f23b00f3-bfd5-4532-9c96-d77ee5af7926"). InnerVolumeSpecName "kube-api-access-m5l8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.184232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.184202 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5l8q\" (UniqueName: \"kubernetes.io/projected/f23b00f3-bfd5-4532-9c96-d77ee5af7926-kube-api-access-m5l8q\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.297108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.297076 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:19.584142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.584103 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:19.657513 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.657468 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:19.740700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.740660 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" path="/var/lib/kubelet/pods/185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b/volumes" Nov 02 23:30:19.852717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.852677 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" Nov 02 23:30:19.853117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.852689 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh" event={"ID":"f23b00f3-bfd5-4532-9c96-d77ee5af7926","Type":"ContainerDied","Data":"af321d690c1d51038f5df59d8981f7bf36cc0b53e35c7eb06c91486eb3524fdf"} Nov 02 23:30:19.853117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:19.852893 4494 scope.go:122] "RemoveContainer" containerID="1bb67e26b9f68e0f65fb98aca525f88cbcc8bd0151bc9d5e68e086652d305966" Nov 02 23:30:20.142277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.142246 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:20.197409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.197385 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:20.201407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.201388 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:20.218354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.218330 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:20.225868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.225844 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894370-9f7kh"] Nov 02 23:30:20.238570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.238547 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\") pod \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " Nov 02 23:30:20.238731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.238716 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\" (UniqueName: \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\") pod \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\" (UID: \"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9\") " Nov 02 23:30:20.238873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.238850 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca" pod "75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" (UID: "75a3dc0a-6807-41aa-a60a-cb96cc9a7df9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.239106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.239091 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.241020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.240995 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6" pod "75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" (UID: "75a3dc0a-6807-41aa-a60a-cb96cc9a7df9"). InnerVolumeSpecName "kube-api-access-d4kh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.339996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.339958 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\" (UniqueName: \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\") pod \"c6962aba-844b-4335-a495-4345f04c687f\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " Nov 02 23:30:20.340237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.340064 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\" (UniqueName: \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\") pod \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " Nov 02 23:30:20.340237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.340113 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\") pod \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\" (UID: \"b83a3a4e-755f-4a3e-8876-473fc6eccb1b\") " Nov 02 23:30:20.340413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.340279 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\") pod \"c6962aba-844b-4335-a495-4345f04c687f\" (UID: \"c6962aba-844b-4335-a495-4345f04c687f\") " Nov 02 23:30:20.340839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.340781 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kh6\" (UniqueName: \"kubernetes.io/projected/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9-kube-api-access-d4kh6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.341293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.341267 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca" pod "b83a3a4e-755f-4a3e-8876-473fc6eccb1b" (UID: "b83a3a4e-755f-4a3e-8876-473fc6eccb1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.341437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.341417 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca" pod "c6962aba-844b-4335-a495-4345f04c687f" (UID: "c6962aba-844b-4335-a495-4345f04c687f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.343160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.343125 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg" pod "c6962aba-844b-4335-a495-4345f04c687f" (UID: "c6962aba-844b-4335-a495-4345f04c687f"). InnerVolumeSpecName "kube-api-access-7qfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.343534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.343514 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx" pod "b83a3a4e-755f-4a3e-8876-473fc6eccb1b" (UID: "b83a3a4e-755f-4a3e-8876-473fc6eccb1b"). InnerVolumeSpecName "kube-api-access-8dwcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.445511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.445399 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfmg\" (UniqueName: \"kubernetes.io/projected/c6962aba-844b-4335-a495-4345f04c687f-kube-api-access-7qfmg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.445511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.445446 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwcx\" (UniqueName: \"kubernetes.io/projected/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-kube-api-access-8dwcx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.445511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.445463 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b83a3a4e-755f-4a3e-8876-473fc6eccb1b-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.445511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.445479 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6962aba-844b-4335-a495-4345f04c687f-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.601333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.601291 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:30:20.601550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.601350 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:30:20.711016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.710679 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:30:20.808316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.808279 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:30:20.864914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.864888 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" Nov 02 23:30:20.865479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.864932 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf" event={"ID":"b83a3a4e-755f-4a3e-8876-473fc6eccb1b","Type":"ContainerDied","Data":"e2a92842311c4b4bec747e0395bf59cd66ca6c929e0bcd03ee4ab66acb73842f"} Nov 02 23:30:20.865479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.865027 4494 scope.go:122] "RemoveContainer" containerID="a9ab7c9325025bfcc79d25d1471e8791cea25d15407ec281c9fe0a5d62f4039f" Nov 02 23:30:20.867675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.867648 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" event={"ID":"75a3dc0a-6807-41aa-a60a-cb96cc9a7df9","Type":"ContainerDied","Data":"60726306223ce545e5614a9034023fef7cf6be248c4fbac016a766a7cca4b518"} Nov 02 23:30:20.867799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.867771 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9" Nov 02 23:30:20.873154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.873132 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" event={"ID":"c6962aba-844b-4335-a495-4345f04c687f","Type":"ContainerDied","Data":"453d153b02f6eafe6b2eba410475b215df9998912d200d5ce2db91722aca9c67"} Nov 02 23:30:20.873289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.873250 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h" Nov 02 23:30:20.883077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.883045 4494 scope.go:122] "RemoveContainer" containerID="ad21ab2513bbbc6913b037edf33ca59517a17989a3cbef30b83f04bfaaeb2b69" Nov 02 23:30:20.900093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:20.900069 4494 scope.go:122] "RemoveContainer" containerID="eff680950943b1323feeb74f8542ba3c4db9e78b9fad0b26f8eb61cddbeff768" Nov 02 23:30:21.065116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.065025 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:21.070421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.070389 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-85efc6-29894370-bvjtf"] Nov 02 23:30:21.071316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.071286 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:21.180759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.180733 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:21.183820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.183795 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d530c9-29894370-98p9h"] Nov 02 23:30:21.249381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.249342 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:21.287599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.287562 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:21.288046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.288019 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:21.288972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.288951 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:30:21.310348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.310319 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:21.315013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.314977 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894370-mcpc9"] Nov 02 23:30:21.499216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.499157 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:21.697869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.697838 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" path="/var/lib/kubelet/pods/75a3dc0a-6807-41aa-a60a-cb96cc9a7df9/volumes" Nov 02 23:30:21.698322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.698307 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" path="/var/lib/kubelet/pods/b83a3a4e-755f-4a3e-8876-473fc6eccb1b/volumes" Nov 02 23:30:21.698701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.698685 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6962aba-844b-4335-a495-4345f04c687f" path="/var/lib/kubelet/pods/c6962aba-844b-4335-a495-4345f04c687f/volumes" Nov 02 23:30:21.699436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.699417 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" path="/var/lib/kubelet/pods/f23b00f3-bfd5-4532-9c96-d77ee5af7926/volumes" Nov 02 23:30:21.705328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:21.705306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:22.437321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:22.437292 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8_33a8408f-d4a1-4d72-ba21-03f33baccd01/payload-processing/0.log" Nov 02 23:30:23.249664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:23.249636 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv_34b131e8-bada-4765-967e-3d20c22c4d96/payload-processing/0.log" Nov 02 23:30:23.481905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:23.481874 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv_34b131e8-bada-4765-967e-3d20c22c4d96/payload-processing/0.log" Nov 02 23:30:24.328190 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:24.328142 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:25.711163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:25.711121 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" podUID="43f3df88-897e-4562-a15a-92fc7a112d24" containerName="payload-processing" containerID="cri-o://b3101c96f73dc52009ea80ba4553700dda87eb3718e722a339f201f9024d4444" gracePeriod=25 Nov 02 23:30:25.809282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:25.809208 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" podUID="bea26092-fb19-476f-8f6d-df7de5c20313" containerName="payload-pre-processing" containerID="cri-o://3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544" gracePeriod=25 Nov 02 23:30:25.915462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:25.915424 4494 generic.go:356] "Generic (PLEG): container finished" podID="43f3df88-897e-4562-a15a-92fc7a112d24" containerID="b3101c96f73dc52009ea80ba4553700dda87eb3718e722a339f201f9024d4444" exitCode=0 Nov 02 23:30:25.915659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:25.915615 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" event={"ID":"43f3df88-897e-4562-a15a-92fc7a112d24","Type":"ContainerDied","Data":"b3101c96f73dc52009ea80ba4553700dda87eb3718e722a339f201f9024d4444"} Nov 02 23:30:25.982859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:25.982834 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:30:26.060358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.060319 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\") pod \"43f3df88-897e-4562-a15a-92fc7a112d24\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " Nov 02 23:30:26.060540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.060463 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\" (UniqueName: \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\") pod \"43f3df88-897e-4562-a15a-92fc7a112d24\" (UID: \"43f3df88-897e-4562-a15a-92fc7a112d24\") " Nov 02 23:30:26.060786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.060759 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume" pod "43f3df88-897e-4562-a15a-92fc7a112d24" (UID: "43f3df88-897e-4562-a15a-92fc7a112d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:26.061031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.061006 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3df88-897e-4562-a15a-92fc7a112d24-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:26.062816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.062786 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr" pod "43f3df88-897e-4562-a15a-92fc7a112d24" (UID: "43f3df88-897e-4562-a15a-92fc7a112d24"). InnerVolumeSpecName "kube-api-access-wcbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:26.081612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.081592 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:30:26.088781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.088759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:26.161500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.161467 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\") pod \"bea26092-fb19-476f-8f6d-df7de5c20313\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " Nov 02 23:30:26.161706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.161520 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\" (UniqueName: \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\") pod \"bea26092-fb19-476f-8f6d-df7de5c20313\" (UID: \"bea26092-fb19-476f-8f6d-df7de5c20313\") " Nov 02 23:30:26.161896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.161880 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbnr\" (UniqueName: \"kubernetes.io/projected/43f3df88-897e-4562-a15a-92fc7a112d24-kube-api-access-wcbnr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:26.161988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.161969 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume" pod "bea26092-fb19-476f-8f6d-df7de5c20313" (UID: "bea26092-fb19-476f-8f6d-df7de5c20313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:26.163882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.163858 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v" pod "bea26092-fb19-476f-8f6d-df7de5c20313" (UID: "bea26092-fb19-476f-8f6d-df7de5c20313"). InnerVolumeSpecName "kube-api-access-js86v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:26.250970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.250873 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:30:26.251292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251266 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:26.251720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251701 4494 cpu_manager.go:414] "containerMap: removing container" podUID="43f3df88-897e-4562-a15a-92fc7a112d24" containerName="payload-processing" Nov 02 23:30:26.251811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251722 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f3df88-897e-4562-a15a-92fc7a112d24" containerName="payload-processing" Nov 02 23:30:26.251811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251736 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.251811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251743 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.251811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251761 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.251811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251768 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251813 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251821 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251836 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251843 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251861 4494 cpu_manager.go:414] "containerMap: removing container" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251869 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251881 4494 cpu_manager.go:414] "containerMap: removing container" podUID="bea26092-fb19-476f-8f6d-df7de5c20313" containerName="payload-pre-processing" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251949 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea26092-fb19-476f-8f6d-df7de5c20313" containerName="payload-pre-processing" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251990 4494 cpu_manager.go:414] "containerMap: removing container" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.251998 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252016 4494 cpu_manager.go:414] "containerMap: removing container" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252023 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252037 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252044 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252056 4494 cpu_manager.go:414] "containerMap: removing container" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252064 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252079 4494 cpu_manager.go:414] "containerMap: removing container" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252086 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.252090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252097 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252103 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252119 4494 cpu_manager.go:414] "containerMap: removing container" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252125 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252287 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252302 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252312 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252322 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252330 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252339 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252348 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252356 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252365 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252376 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f3df88-897e-4562-a15a-92fc7a112d24" containerName="payload-processing" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252384 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252395 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252404 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:26.253043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.252413 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea26092-fb19-476f-8f6d-df7de5c20313" containerName="payload-pre-processing" Nov 02 23:30:26.256441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.256421 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.263084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.263015 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea26092-fb19-476f-8f6d-df7de5c20313-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:26.263084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.263047 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js86v\" (UniqueName: \"kubernetes.io/projected/bea26092-fb19-476f-8f6d-df7de5c20313-kube-api-access-js86v\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:26.364553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364521 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364576 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364658 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364714 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364748 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxh5\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364886 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364926 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.364971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.364948 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.365038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.365014 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.375783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.375756 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:30:26.466324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466289 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466356 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466399 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466452 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466479 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxh5\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466619 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466673 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466713 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466743 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.466953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466907 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.467084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.466960 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.467084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.467054 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.467580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.467548 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.467694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.467611 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.469037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.469014 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.469287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.469271 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.528651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.528589 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxh5\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.533660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.533630 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.571854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.571822 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:26.929502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.929468 4494 generic.go:356] "Generic (PLEG): container finished" podID="bea26092-fb19-476f-8f6d-df7de5c20313" containerID="3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544" exitCode=0 Nov 02 23:30:26.930028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.929529 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" Nov 02 23:30:26.930028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.929564 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" event={"ID":"bea26092-fb19-476f-8f6d-df7de5c20313","Type":"ContainerDied","Data":"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544"} Nov 02 23:30:26.930028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.929628 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w" event={"ID":"bea26092-fb19-476f-8f6d-df7de5c20313","Type":"ContainerDied","Data":"caf0e0c93c38de6c87a049839224e7ed6d1013f767ea935634aa28c6fd4feb50"} Nov 02 23:30:26.930028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.929658 4494 scope.go:122] "RemoveContainer" containerID="3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544" Nov 02 23:30:26.931531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.931498 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" Nov 02 23:30:26.931531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.931523 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph" event={"ID":"43f3df88-897e-4562-a15a-92fc7a112d24","Type":"ContainerDied","Data":"93f2f1c8c4b26b3949686651db4c1899bad2e3937b56a7839e9538ed569eabdd"} Nov 02 23:30:26.941466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.941445 4494 scope.go:122] "RemoveContainer" containerID="3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544" Nov 02 23:30:26.941814 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:26.941796 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544\": container with ID starting with 3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544 not found: ID does not exist" containerID="3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544" Nov 02 23:30:26.941865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.941823 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544"} err="failed to get container status \"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544\": rpc error: code = NotFound desc = could not find container \"3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544\": container with ID starting with 3a3203f91e1016fd4a4525341e3b972b1a4596f9ea428a22c41f936a29680544 not found: ID does not exist" Nov 02 23:30:26.941865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:26.941841 4494 scope.go:122] "RemoveContainer" containerID="b3101c96f73dc52009ea80ba4553700dda87eb3718e722a339f201f9024d4444" Nov 02 23:30:27.004576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.004540 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:30:27.005710 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:27.005682 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0482ab7b_2cd7_4227_99cf_eb78059e5b07.slice/crio-3393a87fe2a8dd9f5d468dbfae4525fca32ffe8fc739b03ac929b1b59f3fc66c WatchSource:0}: Error finding container 3393a87fe2a8dd9f5d468dbfae4525fca32ffe8fc739b03ac929b1b59f3fc66c: Status 404 returned error can't find the container with id 3393a87fe2a8dd9f5d468dbfae4525fca32ffe8fc739b03ac929b1b59f3fc66c Nov 02 23:30:27.008013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.007985 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:27.008094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.008048 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:27.008094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.008077 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:30:27.061352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.061315 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:27.100154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.100124 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:30:27.109694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.108206 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d530c9-554fb8f7d7-pmj4w"] Nov 02 23:30:27.225688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.225598 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:30:27.229812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.229775 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d530c9-b55fc8b75-snvph"] Nov 02 23:30:27.596372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.596278 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:27.699931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.699900 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f3df88-897e-4562-a15a-92fc7a112d24" path="/var/lib/kubelet/pods/43f3df88-897e-4562-a15a-92fc7a112d24/volumes" Nov 02 23:30:27.700371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.700356 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea26092-fb19-476f-8f6d-df7de5c20313" path="/var/lib/kubelet/pods/bea26092-fb19-476f-8f6d-df7de5c20313/volumes" Nov 02 23:30:27.756200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.755965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:27.965536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.965501 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" event={"ID":"0482ab7b-2cd7-4227-99cf-eb78059e5b07","Type":"ContainerStarted","Data":"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0"} Nov 02 23:30:27.965945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:27.965561 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" event={"ID":"0482ab7b-2cd7-4227-99cf-eb78059e5b07","Type":"ContainerStarted","Data":"3393a87fe2a8dd9f5d468dbfae4525fca32ffe8fc739b03ac929b1b59f3fc66c"} Nov 02 23:30:28.095736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:28.094004 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podStartSLOduration=2.09398583 podStartE2EDuration="2.09398583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:28.091814938 +0000 UTC m=+2542.521684800" watchObservedRunningTime="2026-11-02 23:30:28.09398583 +0000 UTC m=+2542.523855691" Nov 02 23:30:28.156254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:28.151790 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:28.571978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:28.571948 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:28.573620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:28.573589 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:28.573748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:28.573642 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:29.379123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:29.379086 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:29.572475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:29.572440 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:29.572664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:29.572508 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:30.572467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:30.572433 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:30.572987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:30.572505 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:31.572688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:31.572654 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:31.573209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:31.572706 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:32.573442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:32.573406 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:32.573952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:32.573478 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:33.573013 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:33.572975 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 02 23:30:33.573225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:33.573074 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 02 23:30:34.330590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:34.330555 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:34.794647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:34.794619 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:34.794970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:34.794943 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:34.795996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:34.795975 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:30:36.281462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:36.281425 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 23:30:36.281966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:36.281696 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" containerID="cri-o://17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff" gracePeriod=30 Nov 02 23:30:37.272741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:37.272703 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6gqk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:30:37.272922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:37.272767 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:30:39.185282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:39.185245 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:40.000576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.000544 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l"] Nov 02 23:30:40.001101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001088 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:40.001152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001103 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:40.001152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001116 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:40.001152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001121 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b00f3-bfd5-4532-9c96-d77ee5af7926" containerName="cleanup" Nov 02 23:30:40.001152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001149 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:40.001297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001154 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83a3a4e-755f-4a3e-8876-473fc6eccb1b" containerName="cleanup" Nov 02 23:30:40.001297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001262 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6962aba-844b-4335-a495-4345f04c687f" containerName="cleanup" Nov 02 23:30:40.001297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001270 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a3dc0a-6807-41aa-a60a-cb96cc9a7df9" containerName="cleanup" Nov 02 23:30:40.001297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.001276 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="185bd76d-5c8e-4dc8-b942-e1fb0fb94a3b" containerName="cleanup" Nov 02 23:30:40.005010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.004994 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.071539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.071506 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:30:40.071881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.071864 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:40.128330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.128294 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l"] Nov 02 23:30:40.181454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.181417 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.181647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.181470 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfggv\" (UniqueName: \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.181647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.181570 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.283311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.283224 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.283311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.283267 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfggv\" (UniqueName: \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.283311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.283306 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.283871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.283850 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.286037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.286006 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.345524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.345490 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfggv\" (UniqueName: \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\") pod \"maas-api-revoke-sub-d616dfa4-snx2l\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.619475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.619394 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:40.996460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:40.996430 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l"] Nov 02 23:30:40.997046 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:40.997018 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dff5711_32f8_421c_9115_d47205992f27.slice/crio-617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9 WatchSource:0}: Error finding container 617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9: Status 404 returned error can't find the container with id 617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9 Nov 02 23:30:41.048773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.048744 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" event={"ID":"2dff5711-32f8-421c-9115-d47205992f27","Type":"ContainerStarted","Data":"617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9"} Nov 02 23:30:41.595884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.595849 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 23:30:41.604512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604482 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.604697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604560 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.604697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604609 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.604697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604687 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.604899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604740 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.604899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604813 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:41.604899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604824 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.605116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.604913 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.605241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605164 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:41.605322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605259 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:41.605322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605287 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:41.605322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605299 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:41.605608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605586 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.605716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.605697 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\") pod \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\" (UID: \"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c\") " Nov 02 23:30:41.606210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.606187 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.606337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.606215 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.606337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.606231 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.606337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.606247 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.606337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.606262 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.607645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.607611 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:30:41.607767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.607682 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "kube-api-access-sxp9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:41.608035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.608005 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:41.608717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.608694 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy" pod "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" (UID: "85c2d72c-58a3-44cb-b0a9-12a9c3ac493c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:30:41.707069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.707030 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.707069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.707060 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.707069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.707072 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxp9m\" (UniqueName: \"kubernetes.io/projected/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-kube-api-access-sxp9m\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:41.707400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:41.707084 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:42.061433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.061395 4494 generic.go:356] "Generic (PLEG): container finished" podID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerID="17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff" exitCode=0 Nov 02 23:30:42.061890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.061861 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" event={"ID":"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c","Type":"ContainerDied","Data":"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff"} Nov 02 23:30:42.066260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.062077 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" event={"ID":"85c2d72c-58a3-44cb-b0a9-12a9c3ac493c","Type":"ContainerDied","Data":"672cc8448eaa2fd8010f1cddd5b8590c171904bebe85b85dc4861cded28bafe0"} Nov 02 23:30:42.066260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.062118 4494 scope.go:122] "RemoveContainer" containerID="17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff" Nov 02 23:30:42.066260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.062314 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7" Nov 02 23:30:42.070340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.070267 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" event={"ID":"2dff5711-32f8-421c-9115-d47205992f27","Type":"ContainerStarted","Data":"a2a65a80976fcc8608e89dcd33dfa9a4bc9e6f0c0f36e4b16955f85864127f21"} Nov 02 23:30:42.082213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.082159 4494 scope.go:122] "RemoveContainer" containerID="17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff" Nov 02 23:30:42.083080 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:42.083050 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff\": container with ID starting with 17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff not found: ID does not exist" containerID="17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff" Nov 02 23:30:42.083212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.083094 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff"} err="failed to get container status \"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff\": rpc error: code = NotFound desc = could not find container \"17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff\": container with ID starting with 17bc6d7c571260182b914e71e0966f6070f331077192b82f37ffa60ea68051ff not found: ID does not exist" Nov 02 23:30:42.242849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.242778 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" podStartSLOduration=3.242756865 podStartE2EDuration="3.242756865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:42.23695352 +0000 UTC m=+2556.666823382" watchObservedRunningTime="2026-11-02 23:30:42.242756865 +0000 UTC m=+2556.672626727" Nov 02 23:30:42.357255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.356982 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 23:30:42.359770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:42.359747 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6gqk7"] Nov 02 23:30:43.078281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:43.078249 4494 generic.go:356] "Generic (PLEG): container finished" podID="2dff5711-32f8-421c-9115-d47205992f27" containerID="a2a65a80976fcc8608e89dcd33dfa9a4bc9e6f0c0f36e4b16955f85864127f21" exitCode=0 Nov 02 23:30:43.078683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:43.078319 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" event={"ID":"2dff5711-32f8-421c-9115-d47205992f27","Type":"ContainerDied","Data":"a2a65a80976fcc8608e89dcd33dfa9a4bc9e6f0c0f36e4b16955f85864127f21"} Nov 02 23:30:43.694341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:43.694304 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" path="/var/lib/kubelet/pods/85c2d72c-58a3-44cb-b0a9-12a9c3ac493c/volumes" Nov 02 23:30:44.240274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.240244 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:44.336944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.336866 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\") pod \"2dff5711-32f8-421c-9115-d47205992f27\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " Nov 02 23:30:44.336944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.336917 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\") pod \"2dff5711-32f8-421c-9115-d47205992f27\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " Nov 02 23:30:44.337234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.337057 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\" (UniqueName: \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\") pod \"2dff5711-32f8-421c-9115-d47205992f27\" (UID: \"2dff5711-32f8-421c-9115-d47205992f27\") " Nov 02 23:30:44.337399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.337374 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca" pod "2dff5711-32f8-421c-9115-d47205992f27" (UID: "2dff5711-32f8-421c-9115-d47205992f27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:44.337622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.337602 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dff5711-32f8-421c-9115-d47205992f27-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.339557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.339537 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv" pod "2dff5711-32f8-421c-9115-d47205992f27" (UID: "2dff5711-32f8-421c-9115-d47205992f27"). InnerVolumeSpecName "kube-api-access-jfggv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:44.340305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.340279 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca" pod "2dff5711-32f8-421c-9115-d47205992f27" (UID: "2dff5711-32f8-421c-9115-d47205992f27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:44.418071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.418039 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:44.438745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.438722 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfggv\" (UniqueName: \"kubernetes.io/projected/2dff5711-32f8-421c-9115-d47205992f27-kube-api-access-jfggv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:44.438745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:44.438744 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dff5711-32f8-421c-9115-d47205992f27-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:45.092580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:45.092550 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" Nov 02 23:30:45.092749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:45.092588 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l" event={"ID":"2dff5711-32f8-421c-9115-d47205992f27","Type":"ContainerDied","Data":"617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9"} Nov 02 23:30:45.092749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:45.092645 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617d79b8fc4cbf0461de5b50fc16f8512a386fb9fe7140b1186c1b2669f898e9" Nov 02 23:30:47.561715 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:47.561623 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:30:47.929008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:47.928967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:48.417660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:48.417620 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:49.526189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526141 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:30:49.526777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526762 4494 cpu_manager.go:414] "containerMap: removing container" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" Nov 02 23:30:49.526826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526779 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" Nov 02 23:30:49.526826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526795 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2dff5711-32f8-421c-9115-d47205992f27" containerName="revoke-keys" Nov 02 23:30:49.526826 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526800 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dff5711-32f8-421c-9115-d47205992f27" containerName="revoke-keys" Nov 02 23:30:49.526919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526909 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c2d72c-58a3-44cb-b0a9-12a9c3ac493c" containerName="istio-proxy" Nov 02 23:30:49.526953 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.526922 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dff5711-32f8-421c-9115-d47205992f27" containerName="revoke-keys" Nov 02 23:30:49.530556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.530540 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.587525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.587493 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:30:49.616234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.614128 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.616234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.614281 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4djl\" (UniqueName: \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.634467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.634424 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-818fe8\"" Nov 02 23:30:49.634613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.634537 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-818fe8-dockercfg-znrgv\"" Nov 02 23:30:49.716065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.716038 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.716262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.716131 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4djl\" (UniqueName: \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.716792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.716771 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.778951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.778878 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4djl\" (UniqueName: \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\") pod \"payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:49.845359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:49.845327 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:50.222948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.222915 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:30:50.223818 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:50.223792 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdb853b_39fd_481c_b7a6_4ef14e32a5c5.slice/crio-690b9a5a08e4351696f11d55e1b9a74362907840176df4515c84ec886acd2c74 WatchSource:0}: Error finding container 690b9a5a08e4351696f11d55e1b9a74362907840176df4515c84ec886acd2c74: Status 404 returned error can't find the container with id 690b9a5a08e4351696f11d55e1b9a74362907840176df4515c84ec886acd2c74 Nov 02 23:30:50.442835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.442805 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:30:50.443054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.443027 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" podUID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" containerName="maas-api" containerID="cri-o://67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41" gracePeriod=30 Nov 02 23:30:50.545591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.545495 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:50.751353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.751320 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:30:50.830625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.830546 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\") pod \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " Nov 02 23:30:50.830625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.830584 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\") pod \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " Nov 02 23:30:50.830850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.830630 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") pod \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " Nov 02 23:30:50.830850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.830690 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\" (UniqueName: \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\") pod \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\" (UID: \"69578c7c-0d3c-4cc4-ab99-76be50f0cb36\") " Nov 02 23:30:50.830969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.830934 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca" pod "69578c7c-0d3c-4cc4-ab99-76be50f0cb36" (UID: "69578c7c-0d3c-4cc4-ab99-76be50f0cb36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:50.831263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.831241 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:50.832941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.832916 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs" pod "69578c7c-0d3c-4cc4-ab99-76be50f0cb36" (UID: "69578c7c-0d3c-4cc4-ab99-76be50f0cb36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:50.833055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.832993 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5" pod "69578c7c-0d3c-4cc4-ab99-76be50f0cb36" (UID: "69578c7c-0d3c-4cc4-ab99-76be50f0cb36"). InnerVolumeSpecName "kube-api-access-nscb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:50.833096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.833078 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls" pod "69578c7c-0d3c-4cc4-ab99-76be50f0cb36" (UID: "69578c7c-0d3c-4cc4-ab99-76be50f0cb36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:50.932580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.932540 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:50.932580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.932578 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:50.932840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:50.932596 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscb5\" (UniqueName: \"kubernetes.io/projected/69578c7c-0d3c-4cc4-ab99-76be50f0cb36-kube-api-access-nscb5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:51.131924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.131830 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" event={"ID":"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5","Type":"ContainerStarted","Data":"ff2c5e05e03d2091e9c85778804a5ba045f86ff21bf3ae4323528f0ca4c7462c"} Nov 02 23:30:51.131924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.131894 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" event={"ID":"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5","Type":"ContainerStarted","Data":"690b9a5a08e4351696f11d55e1b9a74362907840176df4515c84ec886acd2c74"} Nov 02 23:30:51.131924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.131923 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:30:51.133640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.133611 4494 generic.go:356] "Generic (PLEG): container finished" podID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" containerID="67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41" exitCode=0 Nov 02 23:30:51.133783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.133654 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" Nov 02 23:30:51.133783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.133681 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" event={"ID":"69578c7c-0d3c-4cc4-ab99-76be50f0cb36","Type":"ContainerDied","Data":"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41"} Nov 02 23:30:51.133783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.133726 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6" event={"ID":"69578c7c-0d3c-4cc4-ab99-76be50f0cb36","Type":"ContainerDied","Data":"80f2cb8797f54ab05a04f80855096b5ac309e1f1c21841d2fd8d80776466874c"} Nov 02 23:30:51.133783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.133753 4494 scope.go:122] "RemoveContainer" containerID="67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41" Nov 02 23:30:51.146350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.146328 4494 scope.go:122] "RemoveContainer" containerID="67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41" Nov 02 23:30:51.146663 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:30:51.146638 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41\": container with ID starting with 67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41 not found: ID does not exist" containerID="67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41" Nov 02 23:30:51.146738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.146676 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41"} err="failed to get container status \"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41\": rpc error: code = NotFound desc = could not find container \"67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41\": container with ID starting with 67b6388be646c972f0f28b024de1639eb34fe50f537e86820fedf270592ddb41 not found: ID does not exist" Nov 02 23:30:51.248959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.248900 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" podStartSLOduration=2.248880783 podStartE2EDuration="2.248880783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:51.248029727 +0000 UTC m=+2565.677899589" watchObservedRunningTime="2026-11-02 23:30:51.248880783 +0000 UTC m=+2565.678750647" Nov 02 23:30:51.361608 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.361562 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:30:51.365779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.365747 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d530c9-84b89fb466-4qbl6"] Nov 02 23:30:51.694858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:51.694828 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" path="/var/lib/kubelet/pods/69578c7c-0d3c-4cc4-ab99-76be50f0cb36/volumes" Nov 02 23:30:52.295099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:52.295056 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:56.111573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.111539 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc"] Nov 02 23:30:56.112159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.112144 4494 cpu_manager.go:414] "containerMap: removing container" podUID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" containerName="maas-api" Nov 02 23:30:56.112257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.112160 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" containerName="maas-api" Nov 02 23:30:56.112309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.112298 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="69578c7c-0d3c-4cc4-ab99-76be50f0cb36" containerName="maas-api" Nov 02 23:30:56.116916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.116900 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.170238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.170208 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:30:56.172880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.172852 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc"] Nov 02 23:30:56.196061 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.196032 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.196234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.196093 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.196288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.196222 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jh2s\" (UniqueName: \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.219619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.219593 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:30:56.297376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.297336 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.297558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.297439 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.297558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.297503 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jh2s\" (UniqueName: \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.298300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.298264 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.300592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.300545 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.360005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.359968 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jh2s\" (UniqueName: \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\") pod \"maas-api-revoke-sub-ef2e2684-dmkkc\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.431052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.431024 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:56.812419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:56.812391 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc"] Nov 02 23:30:56.812572 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:30:56.812495 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca73293b_b51c_4d51_b60b_22a58482f046.slice/crio-28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8 WatchSource:0}: Error finding container 28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8: Status 404 returned error can't find the container with id 28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8 Nov 02 23:30:57.173481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:57.173442 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" event={"ID":"ca73293b-b51c-4d51-b60b-22a58482f046","Type":"ContainerStarted","Data":"0b15f9de71896d511b712aefaacee631cc12ae1c0e4e9d4ccd5a2d23bb8aebc1"} Nov 02 23:30:57.173889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:57.173507 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" event={"ID":"ca73293b-b51c-4d51-b60b-22a58482f046","Type":"ContainerStarted","Data":"28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8"} Nov 02 23:30:57.296183 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:57.296128 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" podStartSLOduration=1.296113847 podStartE2EDuration="1.296113847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:57.293811714 +0000 UTC m=+2571.723681576" watchObservedRunningTime="2026-11-02 23:30:57.296113847 +0000 UTC m=+2571.725983709" Nov 02 23:30:57.372750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:57.372711 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:30:58.181630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:58.181594 4494 generic.go:356] "Generic (PLEG): container finished" podID="ca73293b-b51c-4d51-b60b-22a58482f046" containerID="0b15f9de71896d511b712aefaacee631cc12ae1c0e4e9d4ccd5a2d23bb8aebc1" exitCode=0 Nov 02 23:30:58.182106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:58.181693 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" event={"ID":"ca73293b-b51c-4d51-b60b-22a58482f046","Type":"ContainerDied","Data":"0b15f9de71896d511b712aefaacee631cc12ae1c0e4e9d4ccd5a2d23bb8aebc1"} Nov 02 23:30:59.349875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.349847 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:30:59.459568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.459487 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\" (UniqueName: \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\") pod \"ca73293b-b51c-4d51-b60b-22a58482f046\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " Nov 02 23:30:59.459727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.459591 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\") pod \"ca73293b-b51c-4d51-b60b-22a58482f046\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " Nov 02 23:30:59.459727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.459662 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\") pod \"ca73293b-b51c-4d51-b60b-22a58482f046\" (UID: \"ca73293b-b51c-4d51-b60b-22a58482f046\") " Nov 02 23:30:59.459939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.459914 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca" pod "ca73293b-b51c-4d51-b60b-22a58482f046" (UID: "ca73293b-b51c-4d51-b60b-22a58482f046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:59.460156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.460139 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:59.461766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.461742 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s" pod "ca73293b-b51c-4d51-b60b-22a58482f046" (UID: "ca73293b-b51c-4d51-b60b-22a58482f046"). InnerVolumeSpecName "kube-api-access-8jh2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:59.461886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.461841 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca" pod "ca73293b-b51c-4d51-b60b-22a58482f046" (UID: "ca73293b-b51c-4d51-b60b-22a58482f046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:30:59.561733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.561697 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jh2s\" (UniqueName: \"kubernetes.io/projected/ca73293b-b51c-4d51-b60b-22a58482f046-kube-api-access-8jh2s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:30:59.561733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:30:59.561728 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca73293b-b51c-4d51-b60b-22a58482f046-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:00.203503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.203466 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" event={"ID":"ca73293b-b51c-4d51-b60b-22a58482f046","Type":"ContainerDied","Data":"28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8"} Nov 02 23:31:00.203503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.203505 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc" Nov 02 23:31:00.203730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.203520 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28e1ca868e5000072d3f1eea76b168fcabffb33e4cea46faffa20cd5a5fa91b8" Nov 02 23:31:00.399974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.399946 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:00.548966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.548885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:00.655796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.655750 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck"] Nov 02 23:31:00.656473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.656451 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ca73293b-b51c-4d51-b60b-22a58482f046" containerName="revoke-keys" Nov 02 23:31:00.656473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.656473 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca73293b-b51c-4d51-b60b-22a58482f046" containerName="revoke-keys" Nov 02 23:31:00.656671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.656595 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca73293b-b51c-4d51-b60b-22a58482f046" containerName="revoke-keys" Nov 02 23:31:00.660271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.660249 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.763752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.763714 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:00.763752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.763736 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:00.767155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.767133 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck"] Nov 02 23:31:00.777639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.777604 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.777824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.777801 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gn8w\" (UniqueName: \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.777879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.777868 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.878741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.878649 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.878924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.878792 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gn8w\" (UniqueName: \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.878924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.878864 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.879519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.879483 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.881205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.881146 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.940793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.940763 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gn8w\" (UniqueName: \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\") pod \"maas-api-revoke-sub-35cf3d2c-g66ck\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:00.974478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:00.974445 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:01.353586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:01.353557 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck"] Nov 02 23:31:01.354221 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:01.354188 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc33012_d9d3_49ef_bb76_7ee7904fcd90.slice/crio-adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30 WatchSource:0}: Error finding container adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30: Status 404 returned error can't find the container with id adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30 Nov 02 23:31:01.907528 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:01.907376 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:02.143069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.143038 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:31:02.218283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.218252 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" event={"ID":"bfc33012-d9d3-49ef-bb76-7ee7904fcd90","Type":"ContainerStarted","Data":"9066994d6b458ffa6e144fa55ad42203fa712ffacee234a5b6770dea09ff3040"} Nov 02 23:31:02.218471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.218307 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" event={"ID":"bfc33012-d9d3-49ef-bb76-7ee7904fcd90","Type":"ContainerStarted","Data":"adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30"} Nov 02 23:31:02.350937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.350897 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:31:02.351137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.351118 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" containerID="cri-o://375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19" gracePeriod=30 Nov 02 23:31:02.381399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.381347 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" podStartSLOduration=2.381332224 podStartE2EDuration="2.381332224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:02.379722843 +0000 UTC m=+2576.809592705" watchObservedRunningTime="2026-11-02 23:31:02.381332224 +0000 UTC m=+2576.811202084" Nov 02 23:31:02.391106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:02.391077 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:03.149614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:03.149576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:03.226639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:03.226604 4494 generic.go:356] "Generic (PLEG): container finished" podID="bfc33012-d9d3-49ef-bb76-7ee7904fcd90" containerID="9066994d6b458ffa6e144fa55ad42203fa712ffacee234a5b6770dea09ff3040" exitCode=0 Nov 02 23:31:03.226801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:03.226655 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" event={"ID":"bfc33012-d9d3-49ef-bb76-7ee7904fcd90","Type":"ContainerDied","Data":"9066994d6b458ffa6e144fa55ad42203fa712ffacee234a5b6770dea09ff3040"} Nov 02 23:31:04.378673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.378648 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:04.379091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.378646 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:04.543599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.543511 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\") pod \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " Nov 02 23:31:04.543599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.543590 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\" (UniqueName: \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\") pod \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " Nov 02 23:31:04.543830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.543691 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\") pod \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\" (UID: \"bfc33012-d9d3-49ef-bb76-7ee7904fcd90\") " Nov 02 23:31:04.544482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.544302 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca" pod "bfc33012-d9d3-49ef-bb76-7ee7904fcd90" (UID: "bfc33012-d9d3-49ef-bb76-7ee7904fcd90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:04.546373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.546348 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca" pod "bfc33012-d9d3-49ef-bb76-7ee7904fcd90" (UID: "bfc33012-d9d3-49ef-bb76-7ee7904fcd90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:04.546711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.546692 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w" pod "bfc33012-d9d3-49ef-bb76-7ee7904fcd90" (UID: "bfc33012-d9d3-49ef-bb76-7ee7904fcd90"). InnerVolumeSpecName "kube-api-access-7gn8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:04.644922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.644888 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:04.644922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.644918 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:04.644922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:04.644929 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gn8w\" (UniqueName: \"kubernetes.io/projected/bfc33012-d9d3-49ef-bb76-7ee7904fcd90-kube-api-access-7gn8w\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:05.240679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:05.240643 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" event={"ID":"bfc33012-d9d3-49ef-bb76-7ee7904fcd90","Type":"ContainerDied","Data":"adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30"} Nov 02 23:31:05.240943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:05.240694 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf0223aefc6c809782ee26dce43ddd5e828f4571621fc5ff2fc0578a2b76f30" Nov 02 23:31:05.240943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:05.240656 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck" Nov 02 23:31:05.691962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:05.691923 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:06.367245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.367213 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl"] Nov 02 23:31:06.367856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.367838 4494 cpu_manager.go:414] "containerMap: removing container" podUID="bfc33012-d9d3-49ef-bb76-7ee7904fcd90" containerName="revoke-keys" Nov 02 23:31:06.367951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.367860 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc33012-d9d3-49ef-bb76-7ee7904fcd90" containerName="revoke-keys" Nov 02 23:31:06.367994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.367981 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc33012-d9d3-49ef-bb76-7ee7904fcd90" containerName="revoke-keys" Nov 02 23:31:06.371261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.371245 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.428689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.428654 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl"] Nov 02 23:31:06.465568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.465533 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkqr\" (UniqueName: \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.465827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.465656 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.465827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.465765 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.475003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.474974 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:06.475302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.475285 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:06.566673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.566638 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.566847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.566728 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.566847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.566835 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkqr\" (UniqueName: \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.567454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.567429 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.569276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.569256 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.630124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.630056 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkqr\" (UniqueName: \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\") pod \"maas-api-revoke-sub-5239cddb-zrgwl\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.685130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.685102 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:06.739446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:06.739324 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:07.063122 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:07.063092 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff6cccd_6cab_40ff_a517_894064a57bf3.slice/crio-a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd WatchSource:0}: Error finding container a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd: Status 404 returned error can't find the container with id a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd Nov 02 23:31:07.063122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.063106 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl"] Nov 02 23:31:07.256548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.256514 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" event={"ID":"4ff6cccd-6cab-40ff-a517-894064a57bf3","Type":"ContainerStarted","Data":"a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd"} Nov 02 23:31:07.419297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.419256 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:07.635636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.635613 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:31:07.680222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680189 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680236 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680292 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680402 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680480 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680493 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:31:07.680600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680528 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680536 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:31:07.680600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680546 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680644 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680677 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:31:07.680767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680687 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\") pod \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\" (UID: \"12a9f3e4-08d0-499f-a5a9-b691be3792f3\") " Nov 02 23:31:07.680968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680938 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:31:07.681035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.680971 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:07.681460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.681434 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.681613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.681466 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.681613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.681484 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.681613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.681499 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.681613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.681514 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.682825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.682801 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:07.682931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.682835 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "kube-api-access-hdmjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:07.683023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.683007 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:31:07.683075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.683055 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy" pod "12a9f3e4-08d0-499f-a5a9-b691be3792f3" (UID: "12a9f3e4-08d0-499f-a5a9-b691be3792f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:31:07.782886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.782853 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.782886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.782882 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.782886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.782893 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a9f3e4-08d0-499f-a5a9-b691be3792f3-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.783387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.782906 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmjq\" (UniqueName: \"kubernetes.io/projected/12a9f3e4-08d0-499f-a5a9-b691be3792f3-kube-api-access-hdmjq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:07.929700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.929660 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:31:07.930576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.930540 4494 cpu_manager.go:414] "containerMap: removing container" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:07.930576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.930580 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:07.930821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.930765 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:07.934536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.934515 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.984960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.984915 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.984974 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985095 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985129 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985165 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cxn\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985345 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985449 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985566 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.985619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.985609 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:07.990306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.990278 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2279ad-openshift-default-dockercfg-5kb8f\"" Nov 02 23:31:07.994706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:07.994679 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:31:08.052070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.052043 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092082 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092219 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092262 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092305 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092351 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092433 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092464 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.095226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.092496 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cxn\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.105900 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.115311 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.116191 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.116446 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.116668 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.122204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.117190 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.128198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.123337 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.154606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.154567 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.179972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.179881 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.180277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.180249 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cxn\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\") pod \"e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.249760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.249729 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:08.272870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.272799 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:08.285276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.285244 4494 generic.go:356] "Generic (PLEG): container finished" podID="4ff6cccd-6cab-40ff-a517-894064a57bf3" containerID="2359b82c2e555a5e67fb18cc7a9181784293eb9e893bdd5d7727b635f229c9eb" exitCode=0 Nov 02 23:31:08.285428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.285336 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" event={"ID":"4ff6cccd-6cab-40ff-a517-894064a57bf3","Type":"ContainerDied","Data":"2359b82c2e555a5e67fb18cc7a9181784293eb9e893bdd5d7727b635f229c9eb"} Nov 02 23:31:08.287201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.287158 4494 generic.go:356] "Generic (PLEG): container finished" podID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerID="375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19" exitCode=0 Nov 02 23:31:08.287308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.287226 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerDied","Data":"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19"} Nov 02 23:31:08.287308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.287252 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" Nov 02 23:31:08.287308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.287283 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq" event={"ID":"12a9f3e4-08d0-499f-a5a9-b691be3792f3","Type":"ContainerDied","Data":"ca6843fa6323e765c79d45aa30e326c54ed9196e937d8e1a05c974d1fe9d6e20"} Nov 02 23:31:08.287430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.287316 4494 scope.go:122] "RemoveContainer" containerID="375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19" Nov 02 23:31:08.297660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.297637 4494 scope.go:122] "RemoveContainer" containerID="edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca" Nov 02 23:31:08.307422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.307404 4494 scope.go:122] "RemoveContainer" containerID="375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19" Nov 02 23:31:08.307714 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:31:08.307696 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19\": container with ID starting with 375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19 not found: ID does not exist" containerID="375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19" Nov 02 23:31:08.307781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.307725 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19"} err="failed to get container status \"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19\": rpc error: code = NotFound desc = could not find container \"375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19\": container with ID starting with 375e3a096ffdb1806ffa0b6729c056b073f1ad27029bc4133ee0db3fc68f1e19 not found: ID does not exist" Nov 02 23:31:08.307781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.307744 4494 scope.go:122] "RemoveContainer" containerID="edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca" Nov 02 23:31:08.308053 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:31:08.308032 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca\": container with ID starting with edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca not found: ID does not exist" containerID="edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca" Nov 02 23:31:08.308112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.308061 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca"} err="failed to get container status \"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca\": rpc error: code = NotFound desc = could not find container \"edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca\": container with ID starting with edc7ceb50166246e9252cfdc4d98fffcbfb75b14021bb50aff6041a67bc6b8ca not found: ID does not exist" Nov 02 23:31:08.517393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.517298 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:31:08.521228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.521202 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d530c9-openshift-default-5679f9b669-2tswq"] Nov 02 23:31:08.699342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.699317 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:31:08.703139 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:08.703113 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a5ffc8_7670_4cbe_b4b2_31b312b08fc1.slice/crio-3b2c5bc0feb0461369155fb2ee841ae7850e89473141eb5922a503c6292b7e70 WatchSource:0}: Error finding container 3b2c5bc0feb0461369155fb2ee841ae7850e89473141eb5922a503c6292b7e70: Status 404 returned error can't find the container with id 3b2c5bc0feb0461369155fb2ee841ae7850e89473141eb5922a503c6292b7e70 Nov 02 23:31:08.711694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.711657 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:08.711787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.711750 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:08.711850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:08.711792 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:09.294750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.294712 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerStarted","Data":"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4"} Nov 02 23:31:09.295254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.294763 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerStarted","Data":"3b2c5bc0feb0461369155fb2ee841ae7850e89473141eb5922a503c6292b7e70"} Nov 02 23:31:09.377323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.377280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:09.418394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.418337 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podStartSLOduration=2.418316972 podStartE2EDuration="2.418316972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:09.417580695 +0000 UTC m=+2583.847450555" watchObservedRunningTime="2026-11-02 23:31:09.418316972 +0000 UTC m=+2583.848186833" Nov 02 23:31:09.437384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.437359 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:09.554250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.549605 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\") pod \"4ff6cccd-6cab-40ff-a517-894064a57bf3\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " Nov 02 23:31:09.554250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.549951 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\") pod \"4ff6cccd-6cab-40ff-a517-894064a57bf3\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " Nov 02 23:31:09.554250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.550076 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\" (UniqueName: \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\") pod \"4ff6cccd-6cab-40ff-a517-894064a57bf3\" (UID: \"4ff6cccd-6cab-40ff-a517-894064a57bf3\") " Nov 02 23:31:09.557527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.556795 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca" pod "4ff6cccd-6cab-40ff-a517-894064a57bf3" (UID: "4ff6cccd-6cab-40ff-a517-894064a57bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:09.562191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.558293 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca" pod "4ff6cccd-6cab-40ff-a517-894064a57bf3" (UID: "4ff6cccd-6cab-40ff-a517-894064a57bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:09.573493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.572216 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:31:09.577557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.577033 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr" pod "4ff6cccd-6cab-40ff-a517-894064a57bf3" (UID: "4ff6cccd-6cab-40ff-a517-894064a57bf3"). InnerVolumeSpecName "kube-api-access-qkkqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:09.651598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.651567 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:09.651598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.651593 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkqr\" (UniqueName: \"kubernetes.io/projected/4ff6cccd-6cab-40ff-a517-894064a57bf3-kube-api-access-qkkqr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:09.651598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.651603 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff6cccd-6cab-40ff-a517-894064a57bf3-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:09.694973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.694937 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" path="/var/lib/kubelet/pods/12a9f3e4-08d0-499f-a5a9-b691be3792f3/volumes" Nov 02 23:31:09.878981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:09.878901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:10.058711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.058665 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:10.255227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.253020 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:10.276798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.276606 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:10.311940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.310594 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" Nov 02 23:31:10.312464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.312212 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl" event={"ID":"4ff6cccd-6cab-40ff-a517-894064a57bf3","Type":"ContainerDied","Data":"a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd"} Nov 02 23:31:10.312464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.312272 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5af988528484ceb5b97cc52fc8e27e8c8172fe9f974cbd53c7bc26b7f50afdd" Nov 02 23:31:10.334520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.334489 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:10.422038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:10.421995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:11.346246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:11.336241 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:11.359465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:11.359435 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:31:12.226795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:12.226763 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:12.394525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:12.394492 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:12.547694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:12.547615 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:13.183989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:13.183953 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:15.132129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.132098 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:15.407023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.406931 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b"] Nov 02 23:31:15.407624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407603 4494 cpu_manager.go:414] "containerMap: removing container" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:15.407624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407624 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:15.407803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407667 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4ff6cccd-6cab-40ff-a517-894064a57bf3" containerName="revoke-keys" Nov 02 23:31:15.407803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407675 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff6cccd-6cab-40ff-a517-894064a57bf3" containerName="revoke-keys" Nov 02 23:31:15.407915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407826 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff6cccd-6cab-40ff-a517-894064a57bf3" containerName="revoke-keys" Nov 02 23:31:15.407915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.407850 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a9f3e4-08d0-499f-a5a9-b691be3792f3" containerName="istio-proxy" Nov 02 23:31:15.412703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.412680 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.454905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.454868 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:15.458306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.458282 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtlr\" (UniqueName: \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.458423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.458352 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.458423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.458408 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.466001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.465824 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:15.516555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.516522 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:15.524764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.524722 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b"] Nov 02 23:31:15.586199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.570204 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtlr\" (UniqueName: \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.586199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.570313 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.586199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.570351 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.600196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.593728 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.600196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.594117 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.632102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.627656 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:15.654558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.654528 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtlr\" (UniqueName: \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\") pod \"maas-api-revoke-sub-cfb2403f-hh42b\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.751200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.733122 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:15.794386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.794352 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:15.865948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.865908 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:31:15.872360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.872337 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:15.925452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.925413 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2279ad\"" Nov 02 23:31:15.925626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:15.925609 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:31:16.038299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.023709 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.038299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.023785 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjhj\" (UniqueName: \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.055140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.051383 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:31:16.065279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.056629 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.081201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.080868 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:31:16.110313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.110285 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2279ad-dockercfg-dnt48\"" Nov 02 23:31:16.124520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.124493 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgr7w\" (UniqueName: \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.124646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.124541 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.124646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.124628 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.124744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.124661 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjhj\" (UniqueName: \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.125249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.125233 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.186024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.185996 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjhj\" (UniqueName: \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\") pod \"payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.192352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.192331 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:16.196056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.196038 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b"] Nov 02 23:31:16.196781 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:16.196760 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c431f8c_b020_4579_a3f3_99827e5c79b4.slice/crio-be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272 WatchSource:0}: Error finding container be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272: Status 404 returned error can't find the container with id be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272 Nov 02 23:31:16.225371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.225340 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.225505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.225489 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgr7w\" (UniqueName: \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.226098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.226080 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.287289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.287259 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgr7w\" (UniqueName: \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\") pod \"payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.368278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.368195 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" event={"ID":"9c431f8c-b020-4579-a3f3-99827e5c79b4","Type":"ContainerStarted","Data":"be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272"} Nov 02 23:31:16.382380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.382355 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:16.570849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.570818 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:31:16.571495 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:16.571470 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3c73df_f0d3_4f60_9fc7_10dd090adcce.slice/crio-c681425d2f0ad09885db7cfaba9e2444f242f016cb774ad3ecd79d2f0f0190f2 WatchSource:0}: Error finding container c681425d2f0ad09885db7cfaba9e2444f242f016cb774ad3ecd79d2f0f0190f2: Status 404 returned error can't find the container with id c681425d2f0ad09885db7cfaba9e2444f242f016cb774ad3ecd79d2f0f0190f2 Nov 02 23:31:16.759202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:16.759163 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:31:16.759996 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:16.759966 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode882f79f_d81d_4eb3_afb2_37070f62ddc5.slice/crio-16bd2c9e0a9f3e5a9b3a1d3d2a61a99ae48fb470beb48cefc009226f1821a6c3 WatchSource:0}: Error finding container 16bd2c9e0a9f3e5a9b3a1d3d2a61a99ae48fb470beb48cefc009226f1821a6c3: Status 404 returned error can't find the container with id 16bd2c9e0a9f3e5a9b3a1d3d2a61a99ae48fb470beb48cefc009226f1821a6c3 Nov 02 23:31:17.375666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.375627 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" event={"ID":"0e3c73df-f0d3-4f60-9fc7-10dd090adcce","Type":"ContainerStarted","Data":"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270"} Nov 02 23:31:17.376162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.375682 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" event={"ID":"0e3c73df-f0d3-4f60-9fc7-10dd090adcce","Type":"ContainerStarted","Data":"c681425d2f0ad09885db7cfaba9e2444f242f016cb774ad3ecd79d2f0f0190f2"} Nov 02 23:31:17.376162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.375704 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:17.377252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.377230 4494 generic.go:356] "Generic (PLEG): container finished" podID="9c431f8c-b020-4579-a3f3-99827e5c79b4" containerID="14637b7e281c1de01e2b87713af5072e743b6f350e6a2710c69479cc82236d17" exitCode=0 Nov 02 23:31:17.377357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.377260 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" event={"ID":"9c431f8c-b020-4579-a3f3-99827e5c79b4","Type":"ContainerDied","Data":"14637b7e281c1de01e2b87713af5072e743b6f350e6a2710c69479cc82236d17"} Nov 02 23:31:17.378757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.378738 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" event={"ID":"e882f79f-d81d-4eb3-afb2-37070f62ddc5","Type":"ContainerStarted","Data":"ad92dde55ce24dd21c4b42e438620090c9430d3377f0d9a4b472b504935c253a"} Nov 02 23:31:17.378879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.378778 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" event={"ID":"e882f79f-d81d-4eb3-afb2-37070f62ddc5","Type":"ContainerStarted","Data":"16bd2c9e0a9f3e5a9b3a1d3d2a61a99ae48fb470beb48cefc009226f1821a6c3"} Nov 02 23:31:17.378879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.378807 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:17.496218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.496090 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" podStartSLOduration=2.496074639 podStartE2EDuration="2.496074639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:17.492267691 +0000 UTC m=+2591.922137552" watchObservedRunningTime="2026-11-02 23:31:17.496074639 +0000 UTC m=+2591.925944499" Nov 02 23:31:17.559765 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:31:17.559727 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:31:17.735670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:17.735614 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" podStartSLOduration=2.7356000590000003 podStartE2EDuration="2.735600059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:17.733498657 +0000 UTC m=+2592.163368517" watchObservedRunningTime="2026-11-02 23:31:17.735600059 +0000 UTC m=+2592.165469957" Nov 02 23:31:18.526474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.526450 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:18.561356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.561330 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\" (UniqueName: \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\") pod \"9c431f8c-b020-4579-a3f3-99827e5c79b4\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " Nov 02 23:31:18.561502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.561385 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\") pod \"9c431f8c-b020-4579-a3f3-99827e5c79b4\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " Nov 02 23:31:18.561502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.561453 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\") pod \"9c431f8c-b020-4579-a3f3-99827e5c79b4\" (UID: \"9c431f8c-b020-4579-a3f3-99827e5c79b4\") " Nov 02 23:31:18.561748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.561725 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca" pod "9c431f8c-b020-4579-a3f3-99827e5c79b4" (UID: "9c431f8c-b020-4579-a3f3-99827e5c79b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:18.561981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.561960 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:18.563743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.563721 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca" pod "9c431f8c-b020-4579-a3f3-99827e5c79b4" (UID: "9c431f8c-b020-4579-a3f3-99827e5c79b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:18.563743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.563728 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr" pod "9c431f8c-b020-4579-a3f3-99827e5c79b4" (UID: "9c431f8c-b020-4579-a3f3-99827e5c79b4"). InnerVolumeSpecName "kube-api-access-4vtlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:18.663630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.663595 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtlr\" (UniqueName: \"kubernetes.io/projected/9c431f8c-b020-4579-a3f3-99827e5c79b4-kube-api-access-4vtlr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:18.663789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.663637 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c431f8c-b020-4579-a3f3-99827e5c79b4-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:18.873313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:18.873280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:19.392479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:19.392448 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" Nov 02 23:31:19.392693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:19.392483 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b" event={"ID":"9c431f8c-b020-4579-a3f3-99827e5c79b4","Type":"ContainerDied","Data":"be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272"} Nov 02 23:31:19.392693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:19.392554 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be66d19ba581e2b1e531ed1e32e861c1013f97eb637b68491c64319132f9e272" Nov 02 23:31:20.567877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:20.567838 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:25.636336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:25.636297 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:28.388594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:28.388564 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:31:28.389009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:28.388641 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:31:29.421479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.421446 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt"] Nov 02 23:31:29.422079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.422065 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9c431f8c-b020-4579-a3f3-99827e5c79b4" containerName="revoke-keys" Nov 02 23:31:29.422134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.422081 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c431f8c-b020-4579-a3f3-99827e5c79b4" containerName="revoke-keys" Nov 02 23:31:29.422238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.422227 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c431f8c-b020-4579-a3f3-99827e5c79b4" containerName="revoke-keys" Nov 02 23:31:29.426674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.426656 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.480729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.479898 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:29.480729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.479938 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:29.483895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.483872 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt"] Nov 02 23:31:29.495720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.495692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.495923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.495836 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.495998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.495931 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkxm\" (UniqueName: \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.597136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.597105 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkxm\" (UniqueName: \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.597356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.597218 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.597356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.597291 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.597870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.597846 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.599805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.599783 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.658714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.658680 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkxm\" (UniqueName: \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\") pod \"maas-api-revoke-sub-87c6df38-jlntt\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:29.739725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:29.739647 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:30.114767 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:30.114737 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dc7054_aa78_401f_ab17_bdd975a07cb7.slice/crio-d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a WatchSource:0}: Error finding container d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a: Status 404 returned error can't find the container with id d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a Nov 02 23:31:30.114923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:30.114901 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt"] Nov 02 23:31:30.459935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:30.459908 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" event={"ID":"c3dc7054-aa78-401f-ab17-bdd975a07cb7","Type":"ContainerStarted","Data":"84613dc434492f97e4d676852de5cffc521ac776127f9a0d54b01e4786e3c125"} Nov 02 23:31:30.460303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:30.459957 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" event={"ID":"c3dc7054-aa78-401f-ab17-bdd975a07cb7","Type":"ContainerStarted","Data":"d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a"} Nov 02 23:31:30.579480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:30.579429 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" podStartSLOduration=1.579414447 podStartE2EDuration="1.579414447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:30.575696925 +0000 UTC m=+2605.005566787" watchObservedRunningTime="2026-11-02 23:31:30.579414447 +0000 UTC m=+2605.009284307" Nov 02 23:31:31.467356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:31.467321 4494 generic.go:356] "Generic (PLEG): container finished" podID="c3dc7054-aa78-401f-ab17-bdd975a07cb7" containerID="84613dc434492f97e4d676852de5cffc521ac776127f9a0d54b01e4786e3c125" exitCode=0 Nov 02 23:31:31.467759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:31.467403 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" event={"ID":"c3dc7054-aa78-401f-ab17-bdd975a07cb7","Type":"ContainerDied","Data":"84613dc434492f97e4d676852de5cffc521ac776127f9a0d54b01e4786e3c125"} Nov 02 23:31:32.157714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.157680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:32.633490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.633461 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:32.738474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.738396 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\") pod \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " Nov 02 23:31:32.738474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.738435 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\") pod \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " Nov 02 23:31:32.738662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.738522 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\" (UniqueName: \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\") pod \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\" (UID: \"c3dc7054-aa78-401f-ab17-bdd975a07cb7\") " Nov 02 23:31:32.738857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.738831 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca" pod "c3dc7054-aa78-401f-ab17-bdd975a07cb7" (UID: "c3dc7054-aa78-401f-ab17-bdd975a07cb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:32.740780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.740748 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca" pod "c3dc7054-aa78-401f-ab17-bdd975a07cb7" (UID: "c3dc7054-aa78-401f-ab17-bdd975a07cb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:32.740886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.740809 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm" pod "c3dc7054-aa78-401f-ab17-bdd975a07cb7" (UID: "c3dc7054-aa78-401f-ab17-bdd975a07cb7"). InnerVolumeSpecName "kube-api-access-nbkxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:32.840630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.840595 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:32.840630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.840629 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3dc7054-aa78-401f-ab17-bdd975a07cb7-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:32.840630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:32.840645 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbkxm\" (UniqueName: \"kubernetes.io/projected/c3dc7054-aa78-401f-ab17-bdd975a07cb7-kube-api-access-nbkxm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:33.480625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:33.480597 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" Nov 02 23:31:33.480783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:33.480643 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt" event={"ID":"c3dc7054-aa78-401f-ab17-bdd975a07cb7","Type":"ContainerDied","Data":"d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a"} Nov 02 23:31:33.480783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:33.480690 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64ef35299097ea6841992e49c8a6bd7644db40d081e442e8d18bf0b2db2443a" Nov 02 23:31:37.037407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:37.037347 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:41.795428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.795397 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr"] Nov 02 23:31:41.795970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.795957 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c3dc7054-aa78-401f-ab17-bdd975a07cb7" containerName="revoke-keys" Nov 02 23:31:41.796016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.795971 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dc7054-aa78-401f-ab17-bdd975a07cb7" containerName="revoke-keys" Nov 02 23:31:41.796105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.796096 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dc7054-aa78-401f-ab17-bdd975a07cb7" containerName="revoke-keys" Nov 02 23:31:41.799952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.799934 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.853849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.853814 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:41.896701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.896667 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ww7\" (UniqueName: \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.896913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.896738 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.896913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.896854 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.904617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.904596 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:41.907984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.907963 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr"] Nov 02 23:31:41.998655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.998619 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ww7\" (UniqueName: \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.998846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.998701 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.998846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.998753 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:41.999429 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:41.999392 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:42.001286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.001264 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:42.060030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.059966 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ww7\" (UniqueName: \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\") pod \"maas-api-revoke-sub-d1f9e0ba-mr4tr\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:42.113840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.113811 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:42.153254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.153212 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:42.495296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.495265 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr"] Nov 02 23:31:42.495774 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:42.495746 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60301f04_3eb6_4ea9_bf60_152b7a9426ee.slice/crio-f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a WatchSource:0}: Error finding container f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a: Status 404 returned error can't find the container with id f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a Nov 02 23:31:42.537828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:42.537796 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" event={"ID":"60301f04-3eb6-4ea9-bf60-152b7a9426ee","Type":"ContainerStarted","Data":"f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a"} Nov 02 23:31:43.547494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:43.547458 4494 generic.go:356] "Generic (PLEG): container finished" podID="60301f04-3eb6-4ea9-bf60-152b7a9426ee" containerID="406514dad2e64e11fdab1a4b1c0f880657ad763a04db7992b6d0a3cab16a2f96" exitCode=0 Nov 02 23:31:43.547959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:43.547542 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" event={"ID":"60301f04-3eb6-4ea9-bf60-152b7a9426ee","Type":"ContainerDied","Data":"406514dad2e64e11fdab1a4b1c0f880657ad763a04db7992b6d0a3cab16a2f96"} Nov 02 23:31:44.707709 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.707680 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:44.739524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.739486 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\") pod \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " Nov 02 23:31:44.739524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.739518 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\") pod \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " Nov 02 23:31:44.739772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.739556 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\" (UniqueName: \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\") pod \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\" (UID: \"60301f04-3eb6-4ea9-bf60-152b7a9426ee\") " Nov 02 23:31:44.739900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.739876 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca" pod "60301f04-3eb6-4ea9-bf60-152b7a9426ee" (UID: "60301f04-3eb6-4ea9-bf60-152b7a9426ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:44.740312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.740291 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:44.741843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.741820 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7" pod "60301f04-3eb6-4ea9-bf60-152b7a9426ee" (UID: "60301f04-3eb6-4ea9-bf60-152b7a9426ee"). InnerVolumeSpecName "kube-api-access-87ww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:44.741967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.741944 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca" pod "60301f04-3eb6-4ea9-bf60-152b7a9426ee" (UID: "60301f04-3eb6-4ea9-bf60-152b7a9426ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:44.841584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.841504 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60301f04-3eb6-4ea9-bf60-152b7a9426ee-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:44.841584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.841536 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ww7\" (UniqueName: \"kubernetes.io/projected/60301f04-3eb6-4ea9-bf60-152b7a9426ee-kube-api-access-87ww7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:44.943465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.943434 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m"] Nov 02 23:31:44.944019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.944006 4494 cpu_manager.go:414] "containerMap: removing container" podUID="60301f04-3eb6-4ea9-bf60-152b7a9426ee" containerName="revoke-keys" Nov 02 23:31:44.944072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.944020 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60301f04-3eb6-4ea9-bf60-152b7a9426ee" containerName="revoke-keys" Nov 02 23:31:44.944133 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.944124 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="60301f04-3eb6-4ea9-bf60-152b7a9426ee" containerName="revoke-keys" Nov 02 23:31:44.947599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:44.947577 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.004258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.004225 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m"] Nov 02 23:31:45.044376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.044342 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghqd\" (UniqueName: \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.044546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.044489 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.044611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.044583 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.145294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.145204 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghqd\" (UniqueName: \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.145460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.145321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.145460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.145387 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.146134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.146057 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.147985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.147963 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.207614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.207588 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghqd\" (UniqueName: \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\") pod \"maas-api-revoke-sub-f8361cd2-r5s6m\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.262109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.262076 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:45.561833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.561806 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" Nov 02 23:31:45.562052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.561842 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr" event={"ID":"60301f04-3eb6-4ea9-bf60-152b7a9426ee","Type":"ContainerDied","Data":"f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a"} Nov 02 23:31:45.562052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.561888 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c056a420c97051206ddcb1a892983c239c797d4c91eca04a31d39fced0bb6a" Nov 02 23:31:45.640308 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:45.640280 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a1b41c_d517_4096_8f8d_8625adb02adb.slice/crio-61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d WatchSource:0}: Error finding container 61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d: Status 404 returned error can't find the container with id 61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d Nov 02 23:31:45.641023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:45.640996 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m"] Nov 02 23:31:46.570689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:46.570648 4494 generic.go:356] "Generic (PLEG): container finished" podID="f0a1b41c-d517-4096-8f8d-8625adb02adb" containerID="3fa271a93fb496442ff86448c97ab8f7e43e1e85306858ef19cb44b7825277da" exitCode=0 Nov 02 23:31:46.571162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:46.570737 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" event={"ID":"f0a1b41c-d517-4096-8f8d-8625adb02adb","Type":"ContainerDied","Data":"3fa271a93fb496442ff86448c97ab8f7e43e1e85306858ef19cb44b7825277da"} Nov 02 23:31:46.571162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:46.570791 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" event={"ID":"f0a1b41c-d517-4096-8f8d-8625adb02adb","Type":"ContainerStarted","Data":"61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d"} Nov 02 23:31:47.562793 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:31:47.562753 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:31:47.737668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.737641 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:47.778020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.777968 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\") pod \"f0a1b41c-d517-4096-8f8d-8625adb02adb\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " Nov 02 23:31:47.778260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.778160 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\") pod \"f0a1b41c-d517-4096-8f8d-8625adb02adb\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " Nov 02 23:31:47.778260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.778238 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\" (UniqueName: \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\") pod \"f0a1b41c-d517-4096-8f8d-8625adb02adb\" (UID: \"f0a1b41c-d517-4096-8f8d-8625adb02adb\") " Nov 02 23:31:47.778627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.778595 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca" pod "f0a1b41c-d517-4096-8f8d-8625adb02adb" (UID: "f0a1b41c-d517-4096-8f8d-8625adb02adb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:47.778797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.778779 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:47.780903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.780874 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca" pod "f0a1b41c-d517-4096-8f8d-8625adb02adb" (UID: "f0a1b41c-d517-4096-8f8d-8625adb02adb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:47.781056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.781038 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd" pod "f0a1b41c-d517-4096-8f8d-8625adb02adb" (UID: "f0a1b41c-d517-4096-8f8d-8625adb02adb"). InnerVolumeSpecName "kube-api-access-2ghqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:47.879599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.879519 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghqd\" (UniqueName: \"kubernetes.io/projected/f0a1b41c-d517-4096-8f8d-8625adb02adb-kube-api-access-2ghqd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:47.879599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:47.879550 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a1b41c-d517-4096-8f8d-8625adb02adb-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:48.585909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:48.585880 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" Nov 02 23:31:48.586131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:48.585926 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m" event={"ID":"f0a1b41c-d517-4096-8f8d-8625adb02adb","Type":"ContainerDied","Data":"61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d"} Nov 02 23:31:48.586131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:48.585978 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61fdb102c732f03e4e413705e7ccc80561df91b9c959976361fd07e22d776b1d" Nov 02 23:31:48.679946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:48.679907 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:52.840477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:52.840446 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:31:52.840931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:52.840685 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" podUID="518472cf-4dbe-45d1-a382-e0194c3af639" containerName="maas-api" containerID="cri-o://ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd" gracePeriod=30 Nov 02 23:31:52.944282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:52.944198 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:53.098602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.098538 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:31:53.142313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.142281 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\") pod \"518472cf-4dbe-45d1-a382-e0194c3af639\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " Nov 02 23:31:53.142473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.142333 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") pod \"518472cf-4dbe-45d1-a382-e0194c3af639\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " Nov 02 23:31:53.142562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.142524 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\" (UniqueName: \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\") pod \"518472cf-4dbe-45d1-a382-e0194c3af639\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " Nov 02 23:31:53.142562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.142549 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\") pod \"518472cf-4dbe-45d1-a382-e0194c3af639\" (UID: \"518472cf-4dbe-45d1-a382-e0194c3af639\") " Nov 02 23:31:53.142682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.142641 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca" pod "518472cf-4dbe-45d1-a382-e0194c3af639" (UID: "518472cf-4dbe-45d1-a382-e0194c3af639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:53.143111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.143093 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/518472cf-4dbe-45d1-a382-e0194c3af639-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.144590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.144568 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls" pod "518472cf-4dbe-45d1-a382-e0194c3af639" (UID: "518472cf-4dbe-45d1-a382-e0194c3af639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:53.145371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.145349 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs" pod "518472cf-4dbe-45d1-a382-e0194c3af639" (UID: "518472cf-4dbe-45d1-a382-e0194c3af639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:31:53.145460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.145353 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t" pod "518472cf-4dbe-45d1-a382-e0194c3af639" (UID: "518472cf-4dbe-45d1-a382-e0194c3af639"). InnerVolumeSpecName "kube-api-access-phr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:53.303886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.303743 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phr2t\" (UniqueName: \"kubernetes.io/projected/518472cf-4dbe-45d1-a382-e0194c3af639-kube-api-access-phr2t\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.303886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.303869 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.304245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.303896 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/518472cf-4dbe-45d1-a382-e0194c3af639-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.313750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.313696 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:31:53.314201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.314029 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" podUID="33a8408f-d4a1-4d72-ba21-03f33baccd01" containerName="payload-processing" containerID="cri-o://56e907e639d52cdf2aee6a7ceb8609e4531baf53d588500d11f2eb3b59002baa" gracePeriod=30 Nov 02 23:31:53.360558 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.360474 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:31:53.360717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.360701 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" podUID="fcfbb708-e07d-4dbb-98b8-455683b4d368" containerName="payload-pre-processing" containerID="cri-o://ca6c97c375bfd8a07e92a88279a539e0f62d4cf234d5bd0998484e48fd04c0a8" gracePeriod=30 Nov 02 23:31:53.459694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.459655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:53.607651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.607613 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:53.624760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.624651 4494 generic.go:356] "Generic (PLEG): container finished" podID="518472cf-4dbe-45d1-a382-e0194c3af639" containerID="ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd" exitCode=0 Nov 02 23:31:53.625223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.624909 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" event={"ID":"518472cf-4dbe-45d1-a382-e0194c3af639","Type":"ContainerDied","Data":"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd"} Nov 02 23:31:53.625223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.625017 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" event={"ID":"518472cf-4dbe-45d1-a382-e0194c3af639","Type":"ContainerDied","Data":"fc1e51654600ac87c9b5741daafe3ad02e38b67f6f2c705560239615010ef8da"} Nov 02 23:31:53.625223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.625077 4494 scope.go:122] "RemoveContainer" containerID="ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd" Nov 02 23:31:53.625492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.625264 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n" Nov 02 23:31:53.627747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.627718 4494 generic.go:356] "Generic (PLEG): container finished" podID="33a8408f-d4a1-4d72-ba21-03f33baccd01" containerID="56e907e639d52cdf2aee6a7ceb8609e4531baf53d588500d11f2eb3b59002baa" exitCode=0 Nov 02 23:31:53.627879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.627868 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" event={"ID":"33a8408f-d4a1-4d72-ba21-03f33baccd01","Type":"ContainerDied","Data":"56e907e639d52cdf2aee6a7ceb8609e4531baf53d588500d11f2eb3b59002baa"} Nov 02 23:31:53.630089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.630065 4494 generic.go:356] "Generic (PLEG): container finished" podID="fcfbb708-e07d-4dbb-98b8-455683b4d368" containerID="ca6c97c375bfd8a07e92a88279a539e0f62d4cf234d5bd0998484e48fd04c0a8" exitCode=0 Nov 02 23:31:53.630258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.630137 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" event={"ID":"fcfbb708-e07d-4dbb-98b8-455683b4d368","Type":"ContainerDied","Data":"ca6c97c375bfd8a07e92a88279a539e0f62d4cf234d5bd0998484e48fd04c0a8"} Nov 02 23:31:53.644331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.644303 4494 scope.go:122] "RemoveContainer" containerID="ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd" Nov 02 23:31:53.644960 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:31:53.644931 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd\": container with ID starting with ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd not found: ID does not exist" containerID="ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd" Nov 02 23:31:53.645078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.644971 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd"} err="failed to get container status \"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd\": rpc error: code = NotFound desc = could not find container \"ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd\": container with ID starting with ab9130905f9e5508afc0aded51ee83d9e510470f3166ef6c22047ae8bd3229fd not found: ID does not exist" Nov 02 23:31:53.650777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.650752 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:31:53.668716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.668695 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:31:53.708624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.708601 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\") pod \"fcfbb708-e07d-4dbb-98b8-455683b4d368\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " Nov 02 23:31:53.708783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.708739 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\" (UniqueName: \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\") pod \"33a8408f-d4a1-4d72-ba21-03f33baccd01\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " Nov 02 23:31:53.708865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.708838 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\" (UniqueName: \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\") pod \"fcfbb708-e07d-4dbb-98b8-455683b4d368\" (UID: \"fcfbb708-e07d-4dbb-98b8-455683b4d368\") " Nov 02 23:31:53.708865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.708854 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\") pod \"33a8408f-d4a1-4d72-ba21-03f33baccd01\" (UID: \"33a8408f-d4a1-4d72-ba21-03f33baccd01\") " Nov 02 23:31:53.708993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.708930 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume" pod "fcfbb708-e07d-4dbb-98b8-455683b4d368" (UID: "fcfbb708-e07d-4dbb-98b8-455683b4d368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:53.709186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.709150 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfbb708-e07d-4dbb-98b8-455683b4d368-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.709312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.709292 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume" pod "33a8408f-d4a1-4d72-ba21-03f33baccd01" (UID: "33a8408f-d4a1-4d72-ba21-03f33baccd01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:53.711303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.711275 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7" pod "fcfbb708-e07d-4dbb-98b8-455683b4d368" (UID: "fcfbb708-e07d-4dbb-98b8-455683b4d368"). InnerVolumeSpecName "kube-api-access-jdhf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:53.711417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.711350 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j" pod "33a8408f-d4a1-4d72-ba21-03f33baccd01" (UID: "33a8408f-d4a1-4d72-ba21-03f33baccd01"). InnerVolumeSpecName "kube-api-access-btf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:53.879020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.863189 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btf6j\" (UniqueName: \"kubernetes.io/projected/33a8408f-d4a1-4d72-ba21-03f33baccd01-kube-api-access-btf6j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.881520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.881451 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhf7\" (UniqueName: \"kubernetes.io/projected/fcfbb708-e07d-4dbb-98b8-455683b4d368-kube-api-access-jdhf7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.881520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.881493 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a8408f-d4a1-4d72-ba21-03f33baccd01-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:31:53.976962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.976926 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:31:53.981857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:53.981818 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb1f8-7f45477d99-wr98n"] Nov 02 23:31:54.187009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.186973 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:54.639908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.639823 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" Nov 02 23:31:54.639908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.639856 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8" event={"ID":"33a8408f-d4a1-4d72-ba21-03f33baccd01","Type":"ContainerDied","Data":"cea674dec7ac2c534764162cb24c830c59d66818059f3e28a7662ad0b221e94f"} Nov 02 23:31:54.640143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.639922 4494 scope.go:122] "RemoveContainer" containerID="56e907e639d52cdf2aee6a7ceb8609e4531baf53d588500d11f2eb3b59002baa" Nov 02 23:31:54.641514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.641493 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" event={"ID":"fcfbb708-e07d-4dbb-98b8-455683b4d368","Type":"ContainerDied","Data":"b78f7b2d0625087d159e1a0b0149dbd5e528c7a28261938b3a4b86cd5ff71788"} Nov 02 23:31:54.641647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.641631 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf" Nov 02 23:31:54.651664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.651472 4494 scope.go:122] "RemoveContainer" containerID="ca6c97c375bfd8a07e92a88279a539e0f62d4cf234d5bd0998484e48fd04c0a8" Nov 02 23:31:54.807127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.807078 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:31:54.857894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.857861 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1cb1f8-554484dcc6-5stl8"] Nov 02 23:31:54.921167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.921113 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:31:54.925756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:54.925731 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1cb1f8-78b5569b9f-kdztf"] Nov 02 23:31:55.694623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:55.694593 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a8408f-d4a1-4d72-ba21-03f33baccd01" path="/var/lib/kubelet/pods/33a8408f-d4a1-4d72-ba21-03f33baccd01/volumes" Nov 02 23:31:55.695136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:55.695115 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518472cf-4dbe-45d1-a382-e0194c3af639" path="/var/lib/kubelet/pods/518472cf-4dbe-45d1-a382-e0194c3af639/volumes" Nov 02 23:31:55.695647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:55.695627 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfbb708-e07d-4dbb-98b8-455683b4d368" path="/var/lib/kubelet/pods/fcfbb708-e07d-4dbb-98b8-455683b4d368/volumes" Nov 02 23:31:55.924465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:55.924421 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:56.090807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.078847 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:56.240104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240072 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:31:56.240691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240676 4494 cpu_manager.go:414] "containerMap: removing container" podUID="fcfbb708-e07d-4dbb-98b8-455683b4d368" containerName="payload-pre-processing" Nov 02 23:31:56.240745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240693 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfbb708-e07d-4dbb-98b8-455683b4d368" containerName="payload-pre-processing" Nov 02 23:31:56.240745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240722 4494 cpu_manager.go:414] "containerMap: removing container" podUID="518472cf-4dbe-45d1-a382-e0194c3af639" containerName="maas-api" Nov 02 23:31:56.240745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240727 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518472cf-4dbe-45d1-a382-e0194c3af639" containerName="maas-api" Nov 02 23:31:56.240839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240752 4494 cpu_manager.go:414] "containerMap: removing container" podUID="33a8408f-d4a1-4d72-ba21-03f33baccd01" containerName="payload-processing" Nov 02 23:31:56.240839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240757 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a8408f-d4a1-4d72-ba21-03f33baccd01" containerName="payload-processing" Nov 02 23:31:56.240839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240765 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f0a1b41c-d517-4096-8f8d-8625adb02adb" containerName="revoke-keys" Nov 02 23:31:56.240839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240769 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a1b41c-d517-4096-8f8d-8625adb02adb" containerName="revoke-keys" Nov 02 23:31:56.240982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240842 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="518472cf-4dbe-45d1-a382-e0194c3af639" containerName="maas-api" Nov 02 23:31:56.240982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240854 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a8408f-d4a1-4d72-ba21-03f33baccd01" containerName="payload-processing" Nov 02 23:31:56.240982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240860 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a1b41c-d517-4096-8f8d-8625adb02adb" containerName="revoke-keys" Nov 02 23:31:56.240982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.240867 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfbb708-e07d-4dbb-98b8-455683b4d368" containerName="payload-pre-processing" Nov 02 23:31:56.245803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.245788 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.290998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.290968 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:31:56.295127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.295111 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.345106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.345020 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:31:56.399367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.399334 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:31:56.417340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417301 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417360 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417387 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417423 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417475 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwxr\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417528 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417556 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417600 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417648 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417680 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2t5\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417708 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417728 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417761 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417808 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417870 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417909 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.417966 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.417995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.418001 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.439088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.439059 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:31:56.439328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.439278 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" containerID="cri-o://a09d99079e8304aeaa7aff46b9d7210c391e7c0138a224d993654138c825aa12" gracePeriod=30 Nov 02 23:31:56.475397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.475312 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:56.519039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519005 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwxr\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519047 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519085 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519254 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519310 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519335 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2t5\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519359 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519384 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519422 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519475 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519545 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519591 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519646 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519664 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519688 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.519715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519696 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519771 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519819 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519823 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519845 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519898 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.519906 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.520158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520015 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520297 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520349 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520551 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.520786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520607 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.521028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.520827 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.522233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.522205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.522497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.522479 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.522605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.522586 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.522697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.522677 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.581983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.581947 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.582679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.582028 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwxr\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.582679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.581952 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.582679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.582073 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2t5\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\") pod \"e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:56.609583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.609516 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:56.703200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.682806 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:56.860220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:56.860125 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:57.147787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.147753 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:31:57.148152 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:57.147849 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0ef8d2_5de0_4016_86a6_fada53414ae8.slice/crio-05070ccea9eb094772763325d93f28e7f0ae6d68f10a1fbc792aa999c630ab49 WatchSource:0}: Error finding container 05070ccea9eb094772763325d93f28e7f0ae6d68f10a1fbc792aa999c630ab49: Status 404 returned error can't find the container with id 05070ccea9eb094772763325d93f28e7f0ae6d68f10a1fbc792aa999c630ab49 Nov 02 23:31:57.150441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.150410 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.150537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.150471 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.150537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.150497 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.169185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.169145 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc"] Nov 02 23:31:57.174428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.174410 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.227693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.227654 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:31:57.228463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.228437 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:31:57.231945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.231914 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc"] Nov 02 23:31:57.262012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.261975 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:57.262837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.262801 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:31:57.263967 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:57.263892 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ce74df_ae18_4e93_abfe_e8e200c41b08.slice/crio-dcff14122b3516cdac5e270acf7ec8ae12708e093451f269a7157101ff9abe12 WatchSource:0}: Error finding container dcff14122b3516cdac5e270acf7ec8ae12708e093451f269a7157101ff9abe12: Status 404 returned error can't find the container with id dcff14122b3516cdac5e270acf7ec8ae12708e093451f269a7157101ff9abe12 Nov 02 23:31:57.267324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.266794 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.267324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.266878 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.267324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.266927 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:31:57.346758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.346725 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqmt\" (UniqueName: \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.346905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.346795 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.347019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.346995 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.448509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.448477 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.448703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.448612 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqmt\" (UniqueName: \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.448703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.448669 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.449199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.449145 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.451297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.451274 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.512375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.512337 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqmt\" (UniqueName: \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\") pod \"maas-api-revoke-sub-32992acd-h25dc\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.519045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.519016 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:31:57.717997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.717897 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" event={"ID":"97ce74df-ae18-4e93-abfe-e8e200c41b08","Type":"ContainerStarted","Data":"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3"} Nov 02 23:31:57.717997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.717964 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" event={"ID":"97ce74df-ae18-4e93-abfe-e8e200c41b08","Type":"ContainerStarted","Data":"dcff14122b3516cdac5e270acf7ec8ae12708e093451f269a7157101ff9abe12"} Nov 02 23:31:57.720186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.720135 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" event={"ID":"0a0ef8d2-5de0-4016-86a6-fada53414ae8","Type":"ContainerStarted","Data":"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a"} Nov 02 23:31:57.720307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.720238 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" event={"ID":"0a0ef8d2-5de0-4016-86a6-fada53414ae8","Type":"ContainerStarted","Data":"05070ccea9eb094772763325d93f28e7f0ae6d68f10a1fbc792aa999c630ab49"} Nov 02 23:31:57.862197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.860188 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:31:57.984244 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:31:57.978469 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55d3d24_cce7_4797_8281_f3d9f780c21f.slice/crio-1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818 WatchSource:0}: Error finding container 1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818: Status 404 returned error can't find the container with id 1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818 Nov 02 23:31:57.993696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:57.992033 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podStartSLOduration=1.9920121320000002 podStartE2EDuration="1.992012132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:57.845707197 +0000 UTC m=+2632.275577060" watchObservedRunningTime="2026-11-02 23:31:57.992012132 +0000 UTC m=+2632.421881990" Nov 02 23:31:58.005833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.005775 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podStartSLOduration=2.005758586 podStartE2EDuration="2.005758586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:57.977427746 +0000 UTC m=+2632.407297617" watchObservedRunningTime="2026-11-02 23:31:58.005758586 +0000 UTC m=+2632.435628518" Nov 02 23:31:58.008366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.008346 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc"] Nov 02 23:31:58.113455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.113404 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:58.610632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.610596 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:31:58.611195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.610672 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:31:58.611195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.610878 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" containerID="cri-o://19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756" gracePeriod=30 Nov 02 23:31:58.611934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.611906 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:31:58.612058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.611957 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:31:58.675197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.673783 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:58.905441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.905384 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:31:58.905684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.905479 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:31:58.918103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.918066 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" event={"ID":"c55d3d24-cce7-4797-8281-f3d9f780c21f","Type":"ContainerStarted","Data":"1ca19e37547719919c985e090475938a8714c15565edc4e705fafefeb6ce069a"} Nov 02 23:31:58.918327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:58.918185 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" event={"ID":"c55d3d24-cce7-4797-8281-f3d9f780c21f","Type":"ContainerStarted","Data":"1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818"} Nov 02 23:31:59.063269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.062098 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" podStartSLOduration=2.062079023 podStartE2EDuration="2.062079023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:59.056207502 +0000 UTC m=+2633.486077360" watchObservedRunningTime="2026-11-02 23:31:59.062079023 +0000 UTC m=+2633.491948884" Nov 02 23:31:59.597095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.597056 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:59.610810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.610784 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:31:59.611277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.610829 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:31:59.755201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.747015 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:31:59.874209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.863575 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:31:59.874209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.863634 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:31:59.925577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.925542 4494 generic.go:356] "Generic (PLEG): container finished" podID="c55d3d24-cce7-4797-8281-f3d9f780c21f" containerID="1ca19e37547719919c985e090475938a8714c15565edc4e705fafefeb6ce069a" exitCode=0 Nov 02 23:31:59.926317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:31:59.926290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" event={"ID":"c55d3d24-cce7-4797-8281-f3d9f780c21f","Type":"ContainerDied","Data":"1ca19e37547719919c985e090475938a8714c15565edc4e705fafefeb6ce069a"} Nov 02 23:32:00.305959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.305926 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw"] Nov 02 23:32:00.310183 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.310151 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.390145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.380939 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw"] Nov 02 23:32:00.405222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.404366 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.405222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.404459 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.405222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.404558 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjvb\" (UniqueName: \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.505657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.505557 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.505657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.505640 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.505893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.505749 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjvb\" (UniqueName: \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.506465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.506437 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.509222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.508600 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.583200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.582886 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjvb\" (UniqueName: \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\") pod \"maas-api-revoke-sub-b5f6a640-dg2zw\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.610357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.610326 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:32:00.610504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.610381 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:32:00.624584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.624553 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:00.869215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.864627 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:32:00.869215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:00.864688 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:32:01.091518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.090674 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw"] Nov 02 23:32:01.094398 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:01.094345 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bc166f_6881_4838_a3cd_0b470cd0d2dc.slice/crio-d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030 WatchSource:0}: Error finding container d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030: Status 404 returned error can't find the container with id d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030 Nov 02 23:32:01.264213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.243551 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:32:01.318378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.318344 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\") pod \"c55d3d24-cce7-4797-8281-f3d9f780c21f\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " Nov 02 23:32:01.318585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.318522 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\") pod \"c55d3d24-cce7-4797-8281-f3d9f780c21f\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " Nov 02 23:32:01.318657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.318606 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\" (UniqueName: \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\") pod \"c55d3d24-cce7-4797-8281-f3d9f780c21f\" (UID: \"c55d3d24-cce7-4797-8281-f3d9f780c21f\") " Nov 02 23:32:01.319142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.319102 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca" pod "c55d3d24-cce7-4797-8281-f3d9f780c21f" (UID: "c55d3d24-cce7-4797-8281-f3d9f780c21f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:01.320949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.320922 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt" pod "c55d3d24-cce7-4797-8281-f3d9f780c21f" (UID: "c55d3d24-cce7-4797-8281-f3d9f780c21f"). InnerVolumeSpecName "kube-api-access-cfqmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:01.321016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.320987 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca" pod "c55d3d24-cce7-4797-8281-f3d9f780c21f" (UID: "c55d3d24-cce7-4797-8281-f3d9f780c21f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:01.392200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.388511 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:32:01.392200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.389611 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c55d3d24-cce7-4797-8281-f3d9f780c21f" containerName="revoke-keys" Nov 02 23:32:01.392200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.389635 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55d3d24-cce7-4797-8281-f3d9f780c21f" containerName="revoke-keys" Nov 02 23:32:01.392200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.389860 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55d3d24-cce7-4797-8281-f3d9f780c21f" containerName="revoke-keys" Nov 02 23:32:01.395022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.394239 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.420605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.420573 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:01.420755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.420611 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfqmt\" (UniqueName: \"kubernetes.io/projected/c55d3d24-cce7-4797-8281-f3d9f780c21f-kube-api-access-cfqmt\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:01.420755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.420631 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55d3d24-cce7-4797-8281-f3d9f780c21f-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:01.428915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.428877 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:32:01.462219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.458136 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2279ad-serving-cert\"" Nov 02 23:32:01.521465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.521435 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mw9t\" (UniqueName: \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.521636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.521489 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.521636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.521530 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.521636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.521551 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.615458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.615424 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:32:01.615618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.615520 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:32:01.618674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.618645 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:01.623048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.623021 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mw9t\" (UniqueName: \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.623305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.623286 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.623524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.623508 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.623665 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:01.623614 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-serving-cert: secret "maas-api-e2e-ipp-b-2279ad-serving-cert" not found Nov 02 23:32:01.623665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.623655 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.623867 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:01.623676 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls podName:f128405e-66be-47da-bab5-b86f29836c33 nodeName:}" failed. No retries permitted until 2026-11-02 23:32:02.123654512 +0000 UTC m=+2636.553524370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls") pod "maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" (UID: "f128405e-66be-47da-bab5-b86f29836c33") : secret "maas-api-e2e-ipp-b-2279ad-serving-cert" not found Nov 02 23:32:01.624736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.624711 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.626121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.626097 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.760455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.749858 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mw9t\" (UniqueName: \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:01.803574 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.803537 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:01.888440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.863537 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:32:01.888440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.863588 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:32:01.976208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:01.957080 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:02.102620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.102589 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" event={"ID":"26bc166f-6881-4838-a3cd-0b470cd0d2dc","Type":"ContainerStarted","Data":"de6fe94e819929a5509adcca72839f666d0270f57acca86ef0728c17ed7d404a"} Nov 02 23:32:02.102853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.102833 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" event={"ID":"26bc166f-6881-4838-a3cd-0b470cd0d2dc","Type":"ContainerStarted","Data":"d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030"} Nov 02 23:32:02.104071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.104049 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" Nov 02 23:32:02.104216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.104089 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc" event={"ID":"c55d3d24-cce7-4797-8281-f3d9f780c21f","Type":"ContainerDied","Data":"1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818"} Nov 02 23:32:02.104216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.104136 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc2380df8bacd73daa2996242f3e4ae694a52b9eb77a858f62c08e6adbff818" Nov 02 23:32:02.106551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.106496 4494 generic.go:356] "Generic (PLEG): container finished" podID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerID="a09d99079e8304aeaa7aff46b9d7210c391e7c0138a224d993654138c825aa12" exitCode=0 Nov 02 23:32:02.106646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.106571 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerDied","Data":"a09d99079e8304aeaa7aff46b9d7210c391e7c0138a224d993654138c825aa12"} Nov 02 23:32:02.106646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.106613 4494 scope.go:122] "RemoveContainer" containerID="9054605860a9d6503d94ea930fef2f89bd0e9327b44b8bd6d8fd8b85a2594a3c" Nov 02 23:32:02.110090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.110073 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:32:02.200793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.200760 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201081 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:02.201150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201118 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201211 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201309 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201385 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201425 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:02.201512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201472 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201498 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201544 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.201630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201571 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\") pod \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\" (UID: \"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7\") " Nov 02 23:32:02.202409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201995 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:02.202409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.201998 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:02.202409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202042 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:02.202409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202337 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:02.202713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202500 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.202713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202521 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.202713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202534 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.202713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202552 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.202713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.202564 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.204273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.204239 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:02.204540 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.204514 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:02.204800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.204781 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "kube-api-access-8skhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:02.204853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.204832 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo" pod "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" (UID: "7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:32:02.205126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.205110 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2279ad-65578c484b-gn726\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:02.228150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.228101 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" podStartSLOduration=2.228083837 podStartE2EDuration="2.228083837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:02.224966981 +0000 UTC m=+2636.654836842" watchObservedRunningTime="2026-11-02 23:32:02.228083837 +0000 UTC m=+2636.657953703" Nov 02 23:32:02.304355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.304316 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.304355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.304346 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.304355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.304357 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.304355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.304366 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skhk\" (UniqueName: \"kubernetes.io/projected/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7-kube-api-access-8skhk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:02.314976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.314927 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:02.609997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.609906 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:32:02.609997 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.609972 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:32:02.778561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.777326 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:32:02.786963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.786901 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:32:02.864285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.864131 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:32:02.864591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:02.864540 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:32:03.114594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.114502 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" Nov 02 23:32:03.114861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.114514 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm" event={"ID":"7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7","Type":"ContainerDied","Data":"b9da4916f60bd193ce73ac6362217e406881250b7f9f0abba6f1c3339056e7fe"} Nov 02 23:32:03.115041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.114991 4494 scope.go:122] "RemoveContainer" containerID="a09d99079e8304aeaa7aff46b9d7210c391e7c0138a224d993654138c825aa12" Nov 02 23:32:03.116293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.116269 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" event={"ID":"f128405e-66be-47da-bab5-b86f29836c33","Type":"ContainerStarted","Data":"dae9e84c615b302696e197750e0ac3c3a2d8b17a9cd15274ffd72f0ef6171147"} Nov 02 23:32:03.118310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.118280 4494 generic.go:356] "Generic (PLEG): container finished" podID="26bc166f-6881-4838-a3cd-0b470cd0d2dc" containerID="de6fe94e819929a5509adcca72839f666d0270f57acca86ef0728c17ed7d404a" exitCode=0 Nov 02 23:32:03.118408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.118317 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" event={"ID":"26bc166f-6881-4838-a3cd-0b470cd0d2dc","Type":"ContainerDied","Data":"de6fe94e819929a5509adcca72839f666d0270f57acca86ef0728c17ed7d404a"} Nov 02 23:32:03.488602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.488557 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:32:03.492485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.492462 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1cb1f8-openshift-default-7bccc4474f-4wfmm"] Nov 02 23:32:03.610696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.610664 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:32:03.610876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.610724 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:32:03.623403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.623360 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:03.693001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.692978 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" path="/var/lib/kubelet/pods/7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7/volumes" Nov 02 23:32:03.860612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.860564 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 02 23:32:03.861199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.860643 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 02 23:32:03.930874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:03.930845 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:32:04.025247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025107 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025215 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025256 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025285 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025368 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025408 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025448 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025457 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025527 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025546 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:04.025595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025578 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\") pod \"44e41bd5-3d46-4b8d-9623-709439954870\" (UID: \"44e41bd5-3d46-4b8d-9623-709439954870\") " Nov 02 23:32:04.026275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.025830 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:04.026275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.026069 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e41bd5-3d46-4b8d-9623-709439954870-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.026275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.026085 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.026275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.026095 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.026529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.026388 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:04.026602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.026531 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:04.028408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.028382 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "kube-api-access-bfr29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:04.028560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.028520 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:04.028950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.028930 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:32:04.028950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.028934 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token" pod "44e41bd5-3d46-4b8d-9623-709439954870" (UID: "44e41bd5-3d46-4b8d-9623-709439954870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:04.127241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127206 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127241 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127255 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e41bd5-3d46-4b8d-9623-709439954870-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127270 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127284 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e41bd5-3d46-4b8d-9623-709439954870-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127297 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfr29\" (UniqueName: \"kubernetes.io/projected/44e41bd5-3d46-4b8d-9623-709439954870-kube-api-access-bfr29\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.127475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127397 4494 generic.go:356] "Generic (PLEG): container finished" podID="44e41bd5-3d46-4b8d-9623-709439954870" containerID="19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756" exitCode=0 Nov 02 23:32:04.127744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127499 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" Nov 02 23:32:04.127744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127536 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerDied","Data":"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756"} Nov 02 23:32:04.127744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127590 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8" event={"ID":"44e41bd5-3d46-4b8d-9623-709439954870","Type":"ContainerDied","Data":"4b4936b0554cbd6be096dc46833d45c879e822896f91181b5e71737b103d58a7"} Nov 02 23:32:04.127744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.127610 4494 scope.go:122] "RemoveContainer" containerID="19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756" Nov 02 23:32:04.136740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.136706 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" event={"ID":"f128405e-66be-47da-bab5-b86f29836c33","Type":"ContainerStarted","Data":"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd"} Nov 02 23:32:04.141703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.141678 4494 scope.go:122] "RemoveContainer" containerID="1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc" Nov 02 23:32:04.153122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.153092 4494 scope.go:122] "RemoveContainer" containerID="19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756" Nov 02 23:32:04.153515 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:04.153485 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756\": container with ID starting with 19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756 not found: ID does not exist" containerID="19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756" Nov 02 23:32:04.153590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.153522 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756"} err="failed to get container status \"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756\": rpc error: code = NotFound desc = could not find container \"19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756\": container with ID starting with 19f1f13b16e454b4f7e8cdce2d27a6834cdf51d853aa0ba025ce2618e2eaf756 not found: ID does not exist" Nov 02 23:32:04.153590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.153546 4494 scope.go:122] "RemoveContainer" containerID="1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc" Nov 02 23:32:04.153943 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:04.153921 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc\": container with ID starting with 1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc not found: ID does not exist" containerID="1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc" Nov 02 23:32:04.154014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.153951 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc"} err="failed to get container status \"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc\": rpc error: code = NotFound desc = could not find container \"1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc\": container with ID starting with 1dc4010a6e8c87fecd93e73d1113db96f71485f7c6533c1beaf55a618f8229dc not found: ID does not exist" Nov 02 23:32:04.286229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.286204 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:04.298064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.298024 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:32:04.301621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.301599 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-adfeab-openshift-default-8484964885-m24t8"] Nov 02 23:32:04.329145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.329112 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\" (UniqueName: \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\") pod \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " Nov 02 23:32:04.329442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.329419 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\") pod \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " Nov 02 23:32:04.329775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.329745 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca" pod "26bc166f-6881-4838-a3cd-0b470cd0d2dc" (UID: "26bc166f-6881-4838-a3cd-0b470cd0d2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:04.329907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.329753 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\") pod \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\" (UID: \"26bc166f-6881-4838-a3cd-0b470cd0d2dc\") " Nov 02 23:32:04.330559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.330535 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.332663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.332627 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca" pod "26bc166f-6881-4838-a3cd-0b470cd0d2dc" (UID: "26bc166f-6881-4838-a3cd-0b470cd0d2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:04.332663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.332648 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb" pod "26bc166f-6881-4838-a3cd-0b470cd0d2dc" (UID: "26bc166f-6881-4838-a3cd-0b470cd0d2dc"). InnerVolumeSpecName "kube-api-access-vjjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:04.421386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.421321 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" podStartSLOduration=2.526486189 podStartE2EDuration="3.421301953s" totalImagesPullingTime="894.815764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:04.416257705 +0000 UTC m=+2638.846127572" watchObservedRunningTime="2026-11-02 23:32:04.421301953 +0000 UTC m=+2638.851171816" Nov 02 23:32:04.432241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.432212 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bc166f-6881-4838-a3cd-0b470cd0d2dc-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.432241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.432240 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjvb\" (UniqueName: \"kubernetes.io/projected/26bc166f-6881-4838-a3cd-0b470cd0d2dc-kube-api-access-vjjvb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:04.610797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.610672 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 02 23:32:04.611088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:04.611042 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 02 23:32:05.149072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.149041 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" Nov 02 23:32:05.149742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.149060 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw" event={"ID":"26bc166f-6881-4838-a3cd-0b470cd0d2dc","Type":"ContainerDied","Data":"d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030"} Nov 02 23:32:05.149839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.149783 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ad4ea040d8abd1a00b0bcae78ab4dc3bb9f8dd3cd89c8909faf223d04a1030" Nov 02 23:32:05.152113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.152087 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:05.260713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.260667 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:32:05.267720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.267691 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:32:05.268145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.268120 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:32:05.269075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.269058 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:32:05.614485 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.614458 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:32:05.614759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.614740 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:32:05.615475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.615455 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:32:05.707069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:05.707030 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e41bd5-3d46-4b8d-9623-709439954870" path="/var/lib/kubelet/pods/44e41bd5-3d46-4b8d-9623-709439954870/volumes" Nov 02 23:32:07.750197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:07.748963 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:10.400375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.400345 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:10.503937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.503901 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf"] Nov 02 23:32:10.504550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504524 4494 cpu_manager.go:414] "containerMap: removing container" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:10.504550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504553 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504569 4494 cpu_manager.go:414] "containerMap: removing container" podUID="26bc166f-6881-4838-a3cd-0b470cd0d2dc" containerName="revoke-keys" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504574 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bc166f-6881-4838-a3cd-0b470cd0d2dc" containerName="revoke-keys" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504589 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504594 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504603 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504608 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504691 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504700 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3c57a2-3a63-4fd7-ab9c-32ab486ceec7" containerName="istio-proxy" Nov 02 23:32:10.504830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504710 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:10.504830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504720 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:10.504830 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.504726 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bc166f-6881-4838-a3cd-0b470cd0d2dc" containerName="revoke-keys" Nov 02 23:32:10.517438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.517405 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.566092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.566062 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf"] Nov 02 23:32:10.612010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.611972 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:10.612253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.612232 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:10.616868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.616837 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42lm\" (UniqueName: \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.616988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.616915 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.617057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.617041 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.718064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.718021 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.718303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.718207 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.718466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.718446 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42lm\" (UniqueName: \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.718753 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.718733 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.720745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.720718 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.780584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.780552 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42lm\" (UniqueName: \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\") pod \"maas-api-revoke-sub-b6d0ad79-tz9sf\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:10.831529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:10.831496 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:11.164774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:11.164749 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:32:11.213046 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:11.213012 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32ab03e_6e23_422d_a581_451ac725dcce.slice/crio-7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602 WatchSource:0}: Error finding container 7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602: Status 404 returned error can't find the container with id 7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602 Nov 02 23:32:11.213214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:11.213121 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf"] Nov 02 23:32:11.236191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:11.236145 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" event={"ID":"b32ab03e-6e23-422d-a581-451ac725dcce","Type":"ContainerStarted","Data":"7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602"} Nov 02 23:32:12.244688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:12.244655 4494 generic.go:356] "Generic (PLEG): container finished" podID="b32ab03e-6e23-422d-a581-451ac725dcce" containerID="5c4aa1541f08d54cb6a643fbc10d41adf5602e1d96732f65ec75e8c5d7727052" exitCode=0 Nov 02 23:32:12.245039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:12.244737 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" event={"ID":"b32ab03e-6e23-422d-a581-451ac725dcce","Type":"ContainerDied","Data":"5c4aa1541f08d54cb6a643fbc10d41adf5602e1d96732f65ec75e8c5d7727052"} Nov 02 23:32:12.737742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:12.737700 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:13.403134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.403110 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:13.450032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.450004 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\") pod \"b32ab03e-6e23-422d-a581-451ac725dcce\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " Nov 02 23:32:13.450210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.450143 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\") pod \"b32ab03e-6e23-422d-a581-451ac725dcce\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " Nov 02 23:32:13.450292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.450251 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\" (UniqueName: \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\") pod \"b32ab03e-6e23-422d-a581-451ac725dcce\" (UID: \"b32ab03e-6e23-422d-a581-451ac725dcce\") " Nov 02 23:32:13.450370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.450349 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca" pod "b32ab03e-6e23-422d-a581-451ac725dcce" (UID: "b32ab03e-6e23-422d-a581-451ac725dcce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:13.450675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.450647 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:13.452436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.452410 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm" pod "b32ab03e-6e23-422d-a581-451ac725dcce" (UID: "b32ab03e-6e23-422d-a581-451ac725dcce"). InnerVolumeSpecName "kube-api-access-k42lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:13.452556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.452537 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca" pod "b32ab03e-6e23-422d-a581-451ac725dcce" (UID: "b32ab03e-6e23-422d-a581-451ac725dcce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:13.552600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.552510 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32ab03e-6e23-422d-a581-451ac725dcce-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:13.552600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:13.552535 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42lm\" (UniqueName: \"kubernetes.io/projected/b32ab03e-6e23-422d-a581-451ac725dcce-kube-api-access-k42lm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:14.258406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:14.258375 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" Nov 02 23:32:14.258593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:14.258416 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf" event={"ID":"b32ab03e-6e23-422d-a581-451ac725dcce","Type":"ContainerDied","Data":"7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602"} Nov 02 23:32:14.258593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:14.258469 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd7dc5e23a5058ed92b7eb8bd0535dd59e63cfde681a589d6114b1397850602" Nov 02 23:32:17.147402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.147348 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:17.487277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.487245 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:32:17.487468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.487440 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" podUID="34b131e8-bada-4765-967e-3d20c22c4d96" containerName="payload-processing" containerID="cri-o://e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d" gracePeriod=30 Nov 02 23:32:17.547625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.547562 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:32:17.547801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.547775 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" podUID="25a02255-0c0e-4293-99a4-fc60d4321022" containerName="payload-pre-processing" containerID="cri-o://5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd" gracePeriod=30 Nov 02 23:32:17.560633 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:17.560599 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:32:17.650670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.650620 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:17.799625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.799580 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:17.822670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.822646 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:32:17.859415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.859391 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:32:17.916386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916352 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\") pod \"34b131e8-bada-4765-967e-3d20c22c4d96\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " Nov 02 23:32:17.916573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916401 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\") pod \"25a02255-0c0e-4293-99a4-fc60d4321022\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " Nov 02 23:32:17.916573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916463 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\" (UniqueName: \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\") pod \"34b131e8-bada-4765-967e-3d20c22c4d96\" (UID: \"34b131e8-bada-4765-967e-3d20c22c4d96\") " Nov 02 23:32:17.916573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916554 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\" (UniqueName: \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\") pod \"25a02255-0c0e-4293-99a4-fc60d4321022\" (UID: \"25a02255-0c0e-4293-99a4-fc60d4321022\") " Nov 02 23:32:17.916856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916835 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume" pod "34b131e8-bada-4765-967e-3d20c22c4d96" (UID: "34b131e8-bada-4765-967e-3d20c22c4d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:17.916906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.916874 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume" pod "25a02255-0c0e-4293-99a4-fc60d4321022" (UID: "25a02255-0c0e-4293-99a4-fc60d4321022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:17.917077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.917060 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b131e8-bada-4765-967e-3d20c22c4d96-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.917135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.917079 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a02255-0c0e-4293-99a4-fc60d4321022-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:17.919019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.918999 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p" pod "25a02255-0c0e-4293-99a4-fc60d4321022" (UID: "25a02255-0c0e-4293-99a4-fc60d4321022"). InnerVolumeSpecName "kube-api-access-wh67p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:17.919102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:17.919021 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn" pod "34b131e8-bada-4765-967e-3d20c22c4d96" (UID: "34b131e8-bada-4765-967e-3d20c22c4d96"). InnerVolumeSpecName "kube-api-access-5pbnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:18.021364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.021265 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh67p\" (UniqueName: \"kubernetes.io/projected/25a02255-0c0e-4293-99a4-fc60d4321022-kube-api-access-wh67p\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:18.021364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.021309 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbnn\" (UniqueName: \"kubernetes.io/projected/34b131e8-bada-4765-967e-3d20c22c4d96-kube-api-access-5pbnn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:18.300966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.300883 4494 generic.go:356] "Generic (PLEG): container finished" podID="34b131e8-bada-4765-967e-3d20c22c4d96" containerID="e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d" exitCode=0 Nov 02 23:32:18.300966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.300944 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" Nov 02 23:32:18.301507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.300981 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" event={"ID":"34b131e8-bada-4765-967e-3d20c22c4d96","Type":"ContainerDied","Data":"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d"} Nov 02 23:32:18.301507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.301040 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv" event={"ID":"34b131e8-bada-4765-967e-3d20c22c4d96","Type":"ContainerDied","Data":"fb4e0e83bfd3c8491c26473c77e88f9c85ebaf04d9fbe47cf790944344547441"} Nov 02 23:32:18.301507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.301078 4494 scope.go:122] "RemoveContainer" containerID="e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d" Nov 02 23:32:18.302519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.302472 4494 generic.go:356] "Generic (PLEG): container finished" podID="25a02255-0c0e-4293-99a4-fc60d4321022" containerID="5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd" exitCode=0 Nov 02 23:32:18.302519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.302512 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" Nov 02 23:32:18.302634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.302534 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" event={"ID":"25a02255-0c0e-4293-99a4-fc60d4321022","Type":"ContainerDied","Data":"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd"} Nov 02 23:32:18.302634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.302567 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk" event={"ID":"25a02255-0c0e-4293-99a4-fc60d4321022","Type":"ContainerDied","Data":"6dae6d511a489b6e6a5386ee6c5d4bdbc351b45c98e65837967b1abfe78ae2aa"} Nov 02 23:32:18.312362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.312316 4494 scope.go:122] "RemoveContainer" containerID="e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d" Nov 02 23:32:18.312981 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:18.312915 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d\": container with ID starting with e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d not found: ID does not exist" containerID="e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d" Nov 02 23:32:18.313073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.312960 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d"} err="failed to get container status \"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d\": rpc error: code = NotFound desc = could not find container \"e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d\": container with ID starting with e5e8542545d8a4d01704b6b602aa0285c9383dc17bbce14edecf8a419bfaf11d not found: ID does not exist" Nov 02 23:32:18.313073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.312993 4494 scope.go:122] "RemoveContainer" containerID="5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd" Nov 02 23:32:18.323435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.323413 4494 scope.go:122] "RemoveContainer" containerID="5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd" Nov 02 23:32:18.323719 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:18.323699 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd\": container with ID starting with 5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd not found: ID does not exist" containerID="5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd" Nov 02 23:32:18.323786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.323727 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd"} err="failed to get container status \"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd\": rpc error: code = NotFound desc = could not find container \"5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd\": container with ID starting with 5b6b89ea429afec219471bc51525becb8f1261c17d9d0501be3f871e2144d8bd not found: ID does not exist" Nov 02 23:32:18.470303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.470268 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:32:18.473452 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.473428 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-014ab4-f7fb58c95-vfhmk"] Nov 02 23:32:18.585270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.585193 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:32:18.589251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.589229 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-014ab4-7bcb9c5b64-2bpqv"] Nov 02 23:32:18.940413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:18.940383 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:19.705600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.705565 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a02255-0c0e-4293-99a4-fc60d4321022" path="/var/lib/kubelet/pods/25a02255-0c0e-4293-99a4-fc60d4321022/volumes" Nov 02 23:32:19.706050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.706034 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b131e8-bada-4765-967e-3d20c22c4d96" path="/var/lib/kubelet/pods/34b131e8-bada-4765-967e-3d20c22c4d96/volumes" Nov 02 23:32:19.771325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771291 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2"] Nov 02 23:32:19.771850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771836 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b32ab03e-6e23-422d-a581-451ac725dcce" containerName="revoke-keys" Nov 02 23:32:19.771901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771853 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32ab03e-6e23-422d-a581-451ac725dcce" containerName="revoke-keys" Nov 02 23:32:19.771901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771865 4494 cpu_manager.go:414] "containerMap: removing container" podUID="34b131e8-bada-4765-967e-3d20c22c4d96" containerName="payload-processing" Nov 02 23:32:19.771901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771870 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b131e8-bada-4765-967e-3d20c22c4d96" containerName="payload-processing" Nov 02 23:32:19.771901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771889 4494 cpu_manager.go:414] "containerMap: removing container" podUID="25a02255-0c0e-4293-99a4-fc60d4321022" containerName="payload-pre-processing" Nov 02 23:32:19.771901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771895 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a02255-0c0e-4293-99a4-fc60d4321022" containerName="payload-pre-processing" Nov 02 23:32:19.772059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771920 4494 cpu_manager.go:414] "containerMap: removing container" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:19.772059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.771925 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e41bd5-3d46-4b8d-9623-709439954870" containerName="istio-proxy" Nov 02 23:32:19.772059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.772013 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32ab03e-6e23-422d-a581-451ac725dcce" containerName="revoke-keys" Nov 02 23:32:19.772059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.772021 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b131e8-bada-4765-967e-3d20c22c4d96" containerName="payload-processing" Nov 02 23:32:19.772059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.772029 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a02255-0c0e-4293-99a4-fc60d4321022" containerName="payload-pre-processing" Nov 02 23:32:19.776704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.776687 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.832103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.832073 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2"] Nov 02 23:32:19.844424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.844389 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42t9\" (UniqueName: \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.844627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.844450 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.844627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.844570 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.878246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.878214 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:19.878246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.878230 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:19.945918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.945877 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42t9\" (UniqueName: \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.946116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.945928 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.946116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.945960 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.946818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.946734 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:19.948665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:19.948642 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:20.007960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:20.007889 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42t9\" (UniqueName: \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\") pod \"maas-api-revoke-sub-9672258a-95kn2\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:20.090681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:20.090646 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:20.469281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:20.469251 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2"] Nov 02 23:32:20.469449 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:20.469359 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98e983f_877e_4a73_bc2d_8ccf8b615a31.slice/crio-bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878 WatchSource:0}: Error finding container bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878: Status 404 returned error can't find the container with id bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878 Nov 02 23:32:21.323923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:21.323895 4494 generic.go:356] "Generic (PLEG): container finished" podID="a98e983f-877e-4a73-bc2d-8ccf8b615a31" containerID="4a5c35afbc5ef7729bc3752e923a869934fd5c6e7c093d8536f68a6d1c9c1a47" exitCode=0 Nov 02 23:32:21.324335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:21.323993 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" event={"ID":"a98e983f-877e-4a73-bc2d-8ccf8b615a31","Type":"ContainerDied","Data":"4a5c35afbc5ef7729bc3752e923a869934fd5c6e7c093d8536f68a6d1c9c1a47"} Nov 02 23:32:21.324335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:21.324047 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" event={"ID":"a98e983f-877e-4a73-bc2d-8ccf8b615a31","Type":"ContainerStarted","Data":"bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878"} Nov 02 23:32:22.488800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.488774 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:22.574742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.574712 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\" (UniqueName: \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\") pod \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " Nov 02 23:32:22.574914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.574821 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\") pod \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " Nov 02 23:32:22.574914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.574886 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\") pod \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\" (UID: \"a98e983f-877e-4a73-bc2d-8ccf8b615a31\") " Nov 02 23:32:22.575287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.575264 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca" pod "a98e983f-877e-4a73-bc2d-8ccf8b615a31" (UID: "a98e983f-877e-4a73-bc2d-8ccf8b615a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:22.577255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.577232 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9" pod "a98e983f-877e-4a73-bc2d-8ccf8b615a31" (UID: "a98e983f-877e-4a73-bc2d-8ccf8b615a31"). InnerVolumeSpecName "kube-api-access-t42t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:22.577365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.577326 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca" pod "a98e983f-877e-4a73-bc2d-8ccf8b615a31" (UID: "a98e983f-877e-4a73-bc2d-8ccf8b615a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:22.676409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.676369 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:22.676593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.676436 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98e983f-877e-4a73-bc2d-8ccf8b615a31-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:22.676593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.676452 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42t9\" (UniqueName: \"kubernetes.io/projected/a98e983f-877e-4a73-bc2d-8ccf8b615a31-kube-api-access-t42t9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:22.926648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.926614 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts"] Nov 02 23:32:22.927199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.927151 4494 cpu_manager.go:414] "containerMap: removing container" podUID="a98e983f-877e-4a73-bc2d-8ccf8b615a31" containerName="revoke-keys" Nov 02 23:32:22.927199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.927182 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98e983f-877e-4a73-bc2d-8ccf8b615a31" containerName="revoke-keys" Nov 02 23:32:22.927389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.927297 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98e983f-877e-4a73-bc2d-8ccf8b615a31" containerName="revoke-keys" Nov 02 23:32:22.931354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.931327 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:22.989121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:22.989087 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts"] Nov 02 23:32:23.081550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.081519 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.081726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.081638 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkdw\" (UniqueName: \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.081773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.081730 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.183310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.183205 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkdw\" (UniqueName: \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.183310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.183310 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.183542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.183408 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.184343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.184313 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.185878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.185852 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.245219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.245167 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkdw\" (UniqueName: \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\") pod \"maas-api-revoke-sub-17a12212-76xts\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.246903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.246879 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:23.343457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.342074 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" event={"ID":"a98e983f-877e-4a73-bc2d-8ccf8b615a31","Type":"ContainerDied","Data":"bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878"} Nov 02 23:32:23.343457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.342125 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc53f7c9f9c31887c9ed08f21e1a47195a8e4bd7d04f080f596a693012bc878" Nov 02 23:32:23.343457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.342216 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2" Nov 02 23:32:23.624551 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:23.624520 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd454a_d4f5_43d4_84bf_c74e709fbfc2.slice/crio-393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47 WatchSource:0}: Error finding container 393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47: Status 404 returned error can't find the container with id 393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47 Nov 02 23:32:23.625973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.625934 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts"] Nov 02 23:32:23.856263 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:23.856235 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:24.363311 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:24.363220 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" event={"ID":"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2","Type":"ContainerStarted","Data":"5b92d8a49437dc02358f402f4af56b4c5caee8b12acf486911afa78b9d754174"} Nov 02 23:32:24.363499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:24.363315 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" event={"ID":"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2","Type":"ContainerStarted","Data":"393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47"} Nov 02 23:32:24.483162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:24.483113 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" podStartSLOduration=2.4831001710000002 podStartE2EDuration="2.483100171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:24.480138778 +0000 UTC m=+2658.910008645" watchObservedRunningTime="2026-11-02 23:32:24.483100171 +0000 UTC m=+2658.912970032" Nov 02 23:32:25.372081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:25.372044 4494 generic.go:356] "Generic (PLEG): container finished" podID="b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" containerID="5b92d8a49437dc02358f402f4af56b4c5caee8b12acf486911afa78b9d754174" exitCode=0 Nov 02 23:32:25.372596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:25.372135 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" event={"ID":"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2","Type":"ContainerDied","Data":"5b92d8a49437dc02358f402f4af56b4c5caee8b12acf486911afa78b9d754174"} Nov 02 23:32:26.527821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.527797 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:26.636563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.636474 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\" (UniqueName: \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\") pod \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " Nov 02 23:32:26.636738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.636572 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\") pod \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " Nov 02 23:32:26.636738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.636633 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\") pod \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\" (UID: \"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2\") " Nov 02 23:32:26.637031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.637003 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca" pod "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" (UID: "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:26.638834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.638807 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca" pod "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" (UID: "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:26.638942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.638855 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw" pod "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" (UID: "b6fd454a-d4f5-43d4-84bf-c74e709fbfc2"). InnerVolumeSpecName "kube-api-access-ndkdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:26.738710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.738664 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:26.738710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.738707 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:26.738710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.738718 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndkdw\" (UniqueName: \"kubernetes.io/projected/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2-kube-api-access-ndkdw\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:26.931638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.931601 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:32:26.932436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.932417 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" containerName="revoke-keys" Nov 02 23:32:26.932555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.932439 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" containerName="revoke-keys" Nov 02 23:32:26.932616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.932605 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" containerName="revoke-keys" Nov 02 23:32:26.936702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:26.936678 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043023 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043134 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043292 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043327 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043376 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043429 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbpl\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043480 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043555 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.043677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.043606 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.045730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.045706 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:32:27.144896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.144852 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.144896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.144908 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.144948 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.144964 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145067 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145094 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145137 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145213 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbpl\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145246 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145418 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145551 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145579 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145724 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.145825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.145781 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.147844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.147815 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.148237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.148214 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.206517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.206446 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.206691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.206672 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbpl\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.251900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.251872 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:27.386799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.386767 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" event={"ID":"b6fd454a-d4f5-43d4-84bf-c74e709fbfc2","Type":"ContainerDied","Data":"393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47"} Nov 02 23:32:27.386978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.386809 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts" Nov 02 23:32:27.386978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.386812 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="393708a63d16e04b14ce018cb6db5ff6f03174b8f89ac3b73ad91b3f50bf4a47" Nov 02 23:32:27.568744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.568663 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:27.686538 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:27.686505 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6264907_348a_4d7b_b652_aba5ea29b90b.slice/crio-f5543a75d4e17bdb13e47687c293a16407d8f9b079dbf44d84832115b2b2911e WatchSource:0}: Error finding container f5543a75d4e17bdb13e47687c293a16407d8f9b079dbf44d84832115b2b2911e: Status 404 returned error can't find the container with id f5543a75d4e17bdb13e47687c293a16407d8f9b079dbf44d84832115b2b2911e Nov 02 23:32:27.689018 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.688965 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.689151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.689062 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.689151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.689124 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:27.693545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.693527 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:32:27.725844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:27.725813 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:28.395066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:28.395028 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" event={"ID":"c6264907-348a-4d7b-b652-aba5ea29b90b","Type":"ContainerStarted","Data":"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc"} Nov 02 23:32:28.395292 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:28.395133 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" event={"ID":"c6264907-348a-4d7b-b652-aba5ea29b90b","Type":"ContainerStarted","Data":"f5543a75d4e17bdb13e47687c293a16407d8f9b079dbf44d84832115b2b2911e"} Nov 02 23:32:28.517921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:28.517850 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podStartSLOduration=2.517836306 podStartE2EDuration="2.517836306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:28.517408963 +0000 UTC m=+2662.947278857" watchObservedRunningTime="2026-11-02 23:32:28.517836306 +0000 UTC m=+2662.947706173" Nov 02 23:32:29.093585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.093551 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:29.253198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.252671 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:29.253198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.252882 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:29.255239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.254686 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:29.255239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.254742 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:29.756284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.756243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:29.760220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.760196 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:32:29.760402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:29.760383 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" containerID="cri-o://cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749" gracePeriod=30 Nov 02 23:32:30.255662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:30.255617 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:30.256232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:30.255692 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:31.255197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.254537 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:31.255197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.254600 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:31.321725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.321691 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:31.762116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.762083 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:32:31.767592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.767572 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.822870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.822835 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-94e3ae-openshift-default-dockercfg-qq6j7\"" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965625 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965726 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965771 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965831 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965867 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965893 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sddvv\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965973 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.965996 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:31.994198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:31.966046 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.008907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.005710 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:32.019312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.019218 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.070841 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.070897 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.070954 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.070977 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.070994 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sddvv\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.071046 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.071061 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.071109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.071138 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.071978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.071859 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.072778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.072749 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.073017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.072994 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.073242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.073213 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.073476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.073447 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.077268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.076675 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.077268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.077142 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.135848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.135756 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sddvv\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.159012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.158953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\") pod \"e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.255201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.254591 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:32.255201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.254670 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:32.288259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.286146 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:32.383087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.383051 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:32.859419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.859384 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:32:32.861661 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:32.861632 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f03dff1_7a1b_4b65_8f9f_00443ed3a2ff.slice/crio-e0c32c05ce0eaa4cbc728f1e1e88e6b734829a7d38e30d4471def7abc5b78634 WatchSource:0}: Error finding container e0c32c05ce0eaa4cbc728f1e1e88e6b734829a7d38e30d4471def7abc5b78634: Status 404 returned error can't find the container with id e0c32c05ce0eaa4cbc728f1e1e88e6b734829a7d38e30d4471def7abc5b78634 Nov 02 23:32:32.864150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.864117 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:32.864247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.864215 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:32.864297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:32.864255 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:33.258201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.255742 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:33.258201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.255830 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:33.456969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.456932 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerStarted","Data":"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893"} Nov 02 23:32:33.457486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.456993 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerStarted","Data":"e0c32c05ce0eaa4cbc728f1e1e88e6b734829a7d38e30d4471def7abc5b78634"} Nov 02 23:32:33.623419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.622403 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podStartSLOduration=2.6223822759999997 podStartE2EDuration="2.622382276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:33.621255293 +0000 UTC m=+2668.051125156" watchObservedRunningTime="2026-11-02 23:32:33.622382276 +0000 UTC m=+2668.052252139" Nov 02 23:32:33.695087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.695055 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7"] Nov 02 23:32:33.700509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.700486 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.769416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.769382 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:33.811224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.770671 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:32:33.811433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.791761 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7"] Nov 02 23:32:33.811519 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.804754 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d249p\" (UniqueName: \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.812241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.811615 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:32:33.812241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.811692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.812241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.811797 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.913410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.913365 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d249p\" (UniqueName: \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.913592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.913500 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.913646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.913629 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.914496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.914426 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.916104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.916083 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:33.975472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:33.975432 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d249p\" (UniqueName: \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\") pod \"maas-api-revoke-sub-eb83de46-qzhw7\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:34.015334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.015303 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:34.256693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.256603 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:34.256693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.256662 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:34.393202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.383396 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:34.408193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.403554 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:34.465199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.464315 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7"] Nov 02 23:32:34.473383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.472726 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:34.473383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:34.472831 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:32:34.477801 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:34.477771 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9deb39_7518_4ccb_aa22_7ae9b721a2bb.slice/crio-bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a WatchSource:0}: Error finding container bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a: Status 404 returned error can't find the container with id bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a Nov 02 23:32:35.361200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.268613 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:35.361200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.269023 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Nov 02 23:32:35.361200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.269068 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Nov 02 23:32:35.489214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.483217 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:35.553893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.553471 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:32:35.556805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.556256 4494 generic.go:356] "Generic (PLEG): container finished" podID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerID="cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749" exitCode=0 Nov 02 23:32:35.556805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.556560 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerDied","Data":"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749"} Nov 02 23:32:35.556805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.556616 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" event={"ID":"28f66ece-b910-4cab-8940-19d95bb7e4e2","Type":"ContainerDied","Data":"683e95466d31f5a4bc7b99a7ac1acb7bb70e99898568d04948c69b4c5506a348"} Nov 02 23:32:35.556805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.556649 4494 scope.go:122] "RemoveContainer" containerID="cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749" Nov 02 23:32:35.585205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.577122 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" event={"ID":"be9deb39-7518-4ccb-aa22-7ae9b721a2bb","Type":"ContainerStarted","Data":"34eb2990b005e21d5f4d46caa8f1ce06f6a59a9807aff136f04d68c6b317cd22"} Nov 02 23:32:35.585205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.577199 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" event={"ID":"be9deb39-7518-4ccb-aa22-7ae9b721a2bb","Type":"ContainerStarted","Data":"bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a"} Nov 02 23:32:35.676213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.661420 4494 scope.go:122] "RemoveContainer" containerID="6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86" Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.676871 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.676927 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677027 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677072 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677148 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677219 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677375 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677428 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.677456 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\") pod \"28f66ece-b910-4cab-8940-19d95bb7e4e2\" (UID: \"28f66ece-b910-4cab-8940-19d95bb7e4e2\") " Nov 02 23:32:35.678635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.678429 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:35.694497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.690381 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:35.694497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.694252 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:35.700217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.694890 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:35.700844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.700443 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:35.935248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.899416 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:32:35.935248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.899599 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:35.935248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.899705 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "kube-api-access-rc7zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:35.935248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:35.899798 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy" pod "28f66ece-b910-4cab-8940-19d95bb7e4e2" (UID: "28f66ece-b910-4cab-8940-19d95bb7e4e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:32:36.050606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.050568 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.050837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.050823 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.050947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.050934 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.078878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.051019 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.079199 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.079370 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7zv\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-kube-api-access-rc7zv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.079492 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f66ece-b910-4cab-8940-19d95bb7e4e2-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.079612 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f66ece-b910-4cab-8940-19d95bb7e4e2-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.079713 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f66ece-b910-4cab-8940-19d95bb7e4e2-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:36.079813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.065954 4494 scope.go:122] "RemoveContainer" containerID="cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749" Nov 02 23:32:36.097928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.097849 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" podStartSLOduration=3.09782366 podStartE2EDuration="3.09782366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:35.97721523 +0000 UTC m=+2670.407085093" watchObservedRunningTime="2026-11-02 23:32:36.09782366 +0000 UTC m=+2670.527693522" Nov 02 23:32:36.101159 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:36.101111 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749\": container with ID starting with cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749 not found: ID does not exist" containerID="cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749" Nov 02 23:32:36.101432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.101388 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749"} err="failed to get container status \"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749\": rpc error: code = NotFound desc = could not find container \"cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749\": container with ID starting with cac2b21cb6073f0f8e528862f79b70db2447ec56f173b955a4251392d886d749 not found: ID does not exist" Nov 02 23:32:36.101594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.101578 4494 scope.go:122] "RemoveContainer" containerID="6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86" Nov 02 23:32:36.102400 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:36.102377 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86\": container with ID starting with 6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86 not found: ID does not exist" containerID="6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86" Nov 02 23:32:36.104680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.104637 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86"} err="failed to get container status \"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86\": rpc error: code = NotFound desc = could not find container \"6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86\": container with ID starting with 6a87b79f1d9cb873d642b0c46f7485fb8da2ac6690cf49a2d65ddc6b7c11af86 not found: ID does not exist" Nov 02 23:32:36.320823 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.320660 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:36.320823 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.320728 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:36.611730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.611583 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:36.663209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.662295 4494 generic.go:356] "Generic (PLEG): container finished" podID="be9deb39-7518-4ccb-aa22-7ae9b721a2bb" containerID="34eb2990b005e21d5f4d46caa8f1ce06f6a59a9807aff136f04d68c6b317cd22" exitCode=0 Nov 02 23:32:36.663209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.662433 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" event={"ID":"be9deb39-7518-4ccb-aa22-7ae9b721a2bb","Type":"ContainerDied","Data":"34eb2990b005e21d5f4d46caa8f1ce06f6a59a9807aff136f04d68c6b317cd22"} Nov 02 23:32:36.673220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.665309 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9" Nov 02 23:32:36.677573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.677263 4494 patch_prober.go:29] interesting pod/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:36.677573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:36.677341 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:37.038202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.028766 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:32:37.103200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.097552 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-014ab4-openshift-default-84c5696b9-lnbr9"] Nov 02 23:32:37.292280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.291202 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:37.315609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.315580 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:37.315856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.315832 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:37.453215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.452317 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:37.718276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.718237 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" path="/var/lib/kubelet/pods/28f66ece-b910-4cab-8940-19d95bb7e4e2/volumes" Nov 02 23:32:37.859125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:37.854901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:38.005495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.005470 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:38.094764 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.094717 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\") pod \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " Nov 02 23:32:38.094962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.094815 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\") pod \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " Nov 02 23:32:38.094962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.094951 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\" (UniqueName: \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\") pod \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\" (UID: \"be9deb39-7518-4ccb-aa22-7ae9b721a2bb\") " Nov 02 23:32:38.095100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.095071 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca" pod "be9deb39-7518-4ccb-aa22-7ae9b721a2bb" (UID: "be9deb39-7518-4ccb-aa22-7ae9b721a2bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:38.095478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.095455 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:38.097230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.097204 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p" pod "be9deb39-7518-4ccb-aa22-7ae9b721a2bb" (UID: "be9deb39-7518-4ccb-aa22-7ae9b721a2bb"). InnerVolumeSpecName "kube-api-access-d249p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:38.097421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.097400 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca" pod "be9deb39-7518-4ccb-aa22-7ae9b721a2bb" (UID: "be9deb39-7518-4ccb-aa22-7ae9b721a2bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:38.196910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.196627 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:38.196910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.196658 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d249p\" (UniqueName: \"kubernetes.io/projected/be9deb39-7518-4ccb-aa22-7ae9b721a2bb-kube-api-access-d249p\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:32:38.364258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.363302 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:32:38.364258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.363366 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:32:38.717344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.717314 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" Nov 02 23:32:38.717344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.717335 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7" event={"ID":"be9deb39-7518-4ccb-aa22-7ae9b721a2bb","Type":"ContainerDied","Data":"bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a"} Nov 02 23:32:38.717595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.717386 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee5749fb30227e5c9e9c26ed24fb7f0d19ff953b25279e656784c16c5919f2a" Nov 02 23:32:38.963811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.962703 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:38.963811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.963657 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:38.965369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:38.965346 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:32:40.387319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.387278 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:40.598270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.598140 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:40.805527 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.805490 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:32:40.806967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.806937 4494 cpu_manager.go:414] "containerMap: removing container" podUID="be9deb39-7518-4ccb-aa22-7ae9b721a2bb" containerName="revoke-keys" Nov 02 23:32:40.806967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.806961 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9deb39-7518-4ccb-aa22-7ae9b721a2bb" containerName="revoke-keys" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.806985 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.807001 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.807025 4494 cpu_manager.go:414] "containerMap: removing container" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.807030 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.807112 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:40.807222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.807123 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9deb39-7518-4ccb-aa22-7ae9b721a2bb" containerName="revoke-keys" Nov 02 23:32:40.810303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.810286 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:40.839939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.839901 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvmk\" (UniqueName: \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:40.840498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.840475 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:40.864543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.864516 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:32:40.911923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.911886 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-94e3ae\"" Nov 02 23:32:40.912443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.912417 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-94e3ae-dockercfg-gmq4j\"" Nov 02 23:32:40.966202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.962036 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvmk\" (UniqueName: \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:40.966202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.962127 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:40.966202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:40.963138 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:41.018366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.017270 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:32:41.018617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.018485 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f66ece-b910-4cab-8940-19d95bb7e4e2" containerName="istio-proxy" Nov 02 23:32:41.037859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.037829 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.064700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.064603 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27zf\" (UniqueName: \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.064700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.064657 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.070235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.069020 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:32:41.081201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.079853 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvmk\" (UniqueName: \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\") pod \"payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:41.123819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.123796 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:41.165959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.165909 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27zf\" (UniqueName: \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.166107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.165968 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.167122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.167088 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.250267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.250139 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27zf\" (UniqueName: \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\") pod \"payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.398974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.398898 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:41.559898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.559499 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:32:41.784481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.784446 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" event={"ID":"668c3b82-4242-43eb-946f-78a802c34d4e","Type":"ContainerStarted","Data":"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466"} Nov 02 23:32:41.784678 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.784511 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" event={"ID":"668c3b82-4242-43eb-946f-78a802c34d4e","Type":"ContainerStarted","Data":"897dccb614aeee332e1421dfa5d25a3acd278e9f926da0f32dd1aba54efef720"} Nov 02 23:32:41.784735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.784685 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:41.802922 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:41.802887 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d96b446_1ec0_45a7_9358_a5d9e31ee69e.slice/crio-f7b953752aa3a2591d679ca230426327da9fe669509919cf25a7a1df3f3f8438 WatchSource:0}: Error finding container f7b953752aa3a2591d679ca230426327da9fe669509919cf25a7a1df3f3f8438: Status 404 returned error can't find the container with id f7b953752aa3a2591d679ca230426327da9fe669509919cf25a7a1df3f3f8438 Nov 02 23:32:41.803140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.803124 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:32:41.874453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.874423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:41.914340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:41.914287 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" podStartSLOduration=1.914270556 podStartE2EDuration="1.914270556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:41.911738774 +0000 UTC m=+2676.341608648" watchObservedRunningTime="2026-11-02 23:32:41.914270556 +0000 UTC m=+2676.344140416" Nov 02 23:32:42.093124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.093036 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:32:42.099715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.099687 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.127019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.126990 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:32:42.180554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180513 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.180554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180566 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbz9\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.180842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180628 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.180842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180694 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.180842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180789 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.181017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180911 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.181017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180942 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.181017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.180998 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.181165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.181026 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.282885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.282842 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.283183 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.283140 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.283358 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.283340 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.283477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.283463 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.283711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.283694 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.285530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.285499 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.285720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.285697 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbz9\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.285895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.285880 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.286029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.286012 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.286246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.286226 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.286355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.286331 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.286417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.285764 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.289011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.288985 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.289207 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.289100 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.291875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.291846 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.293969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.293943 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.347848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.347778 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.349437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.349411 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbz9\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\") pod \"e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.434198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.420439 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:42.800316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.800271 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" event={"ID":"6d96b446-1ec0-45a7-9358-a5d9e31ee69e","Type":"ContainerStarted","Data":"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67"} Nov 02 23:32:42.800525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.800333 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" event={"ID":"6d96b446-1ec0-45a7-9358-a5d9e31ee69e","Type":"ContainerStarted","Data":"f7b953752aa3a2591d679ca230426327da9fe669509919cf25a7a1df3f3f8438"} Nov 02 23:32:42.800617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.800533 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:42.886753 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:42.886719 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb12f013_bf28_40bf_ae94_fc775ca78045.slice/crio-d878f9094a2f0c44da185a920269f727a6c105b0499fb4d1a3e446656ac9394b WatchSource:0}: Error finding container d878f9094a2f0c44da185a920269f727a6c105b0499fb4d1a3e446656ac9394b: Status 404 returned error can't find the container with id d878f9094a2f0c44da185a920269f727a6c105b0499fb4d1a3e446656ac9394b Nov 02 23:32:42.888467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.888438 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:32:42.889417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.889388 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:42.889488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.889449 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:42.889488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.889477 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:42.921637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:42.921592 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" podStartSLOduration=2.921578237 podStartE2EDuration="2.921578237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:42.918058484 +0000 UTC m=+2677.347928346" watchObservedRunningTime="2026-11-02 23:32:42.921578237 +0000 UTC m=+2677.351448104" Nov 02 23:32:43.174605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.174572 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 02 23:32:43.179249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.179226 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.204943 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.204902 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbb9\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.204961 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205031 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205092 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205138 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205198 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205244 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205283 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.205427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.205402 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.236588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.235896 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f46b61-openshift-default-dockercfg-98n4w\"" Nov 02 23:32:43.248072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.247999 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 02 23:32:43.254783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.254693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:43.306534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306497 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.306787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306590 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.306787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306647 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.306787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306691 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.306787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306741 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.306787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306783 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.307211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306907 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.307211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.306977 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbb9\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.307211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.307054 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.307401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.307315 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.308226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.308152 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.308494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.308453 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.308632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.308565 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.308889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.308686 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.309908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.309874 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.310029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.309984 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.370402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.370361 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.370581 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.370463 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbb9\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\") pod \"e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.402417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.402149 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:43.505334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.505109 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:43.548920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.548159 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:43.811745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.811650 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" event={"ID":"bb12f013-bf28-40bf-ae94-fc775ca78045","Type":"ContainerStarted","Data":"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a"} Nov 02 23:32:43.811745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.811713 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" event={"ID":"bb12f013-bf28-40bf-ae94-fc775ca78045","Type":"ContainerStarted","Data":"d878f9094a2f0c44da185a920269f727a6c105b0499fb4d1a3e446656ac9394b"} Nov 02 23:32:43.941198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:43.938761 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podStartSLOduration=1.938738332 podStartE2EDuration="1.938738332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:43.935775429 +0000 UTC m=+2678.365645368" watchObservedRunningTime="2026-11-02 23:32:43.938738332 +0000 UTC m=+2678.368608202" Nov 02 23:32:43.989401 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:43.988352 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a81eb1_21f9_4e99_9448_e9c3602b8b16.slice/crio-010523e89e646b03c46024a13ce405feb20f61bd3a7d6de34b16c6ef52c482bd WatchSource:0}: Error finding container 010523e89e646b03c46024a13ce405feb20f61bd3a7d6de34b16c6ef52c482bd: Status 404 returned error can't find the container with id 010523e89e646b03c46024a13ce405feb20f61bd3a7d6de34b16c6ef52c482bd Nov 02 23:32:44.001583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.000273 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:44.001583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.000363 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:44.001583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.000412 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:44.015100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.015065 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 02 23:32:44.161848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.161806 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 02 23:32:44.170147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.170102 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.220974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.220937 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 02 23:32:44.226427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.226403 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-440c60-openshift-default-dockercfg-2cclx\"" Nov 02 23:32:44.255217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.242781 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:44.268914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.268878 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.269136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.269124 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.269261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.269238 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgrp\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.269379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.269361 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.270400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.270353 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.270648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.270628 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.270917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.270897 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.271115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.271080 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.271352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.271315 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424725 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424831 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424863 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgrp\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424900 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424928 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.424963 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.425068 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.425107 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.425162 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.425969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.425751 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.427740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.426598 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.427740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.426818 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.427740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.427459 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.427740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.427711 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.428122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.427758 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:44.463037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.445303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:44.484446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.484417 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:44.484695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.484671 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:44.487839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.487802 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.489592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.489345 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.492483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.492327 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgrp\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.492483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.492437 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\") pod \"e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.787641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.787540 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:44.821894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.821808 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerStarted","Data":"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2"} Nov 02 23:32:44.821894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.821871 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerStarted","Data":"010523e89e646b03c46024a13ce405feb20f61bd3a7d6de34b16c6ef52c482bd"} Nov 02 23:32:44.959810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:44.959737 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podStartSLOduration=1.9597152279999999 podStartE2EDuration="1.959715228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:44.95392391 +0000 UTC m=+2679.383793752" watchObservedRunningTime="2026-11-02 23:32:44.959715228 +0000 UTC m=+2679.389585094" Nov 02 23:32:45.280965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.280929 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 02 23:32:45.282008 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:32:45.281973 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc981b8ca_3c85_4e43_9bfb_d4e2398dec7a.slice/crio-25e5ee3ecf5c715cb84fd6d1012da400fc646a4b60d8b53d3f74f58ed3f6dc11 WatchSource:0}: Error finding container 25e5ee3ecf5c715cb84fd6d1012da400fc646a4b60d8b53d3f74f58ed3f6dc11: Status 404 returned error can't find the container with id 25e5ee3ecf5c715cb84fd6d1012da400fc646a4b60d8b53d3f74f58ed3f6dc11 Nov 02 23:32:45.285355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.285324 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:45.285444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.285395 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:45.285444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.285428 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:45.421345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.421300 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:45.421523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.421368 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:45.505958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.505869 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:45.512248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.512220 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:45.861758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.861712 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerStarted","Data":"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e"} Nov 02 23:32:45.862298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.861949 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:45.862298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.861982 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerStarted","Data":"25e5ee3ecf5c715cb84fd6d1012da400fc646a4b60d8b53d3f74f58ed3f6dc11"} Nov 02 23:32:45.863456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.863433 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:32:45.984742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:45.984680 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podStartSLOduration=1.984663396 podStartE2EDuration="1.984663396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:45.984187995 +0000 UTC m=+2680.414057853" watchObservedRunningTime="2026-11-02 23:32:45.984663396 +0000 UTC m=+2680.414533261" Nov 02 23:32:46.421664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.421629 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:46.421827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.421691 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:46.788545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.788460 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:46.794863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.794838 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:46.869952 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.869873 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:46.870867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:46.870846 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:32:47.421610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.421581 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:47.423605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.421641 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:47.588478 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:32:47.588397 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:32:47.783478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.783396 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:47.878596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.878561 4494 generic.go:356] "Generic (PLEG): container finished" podID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerID="c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4" exitCode=0 Nov 02 23:32:47.879076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.878627 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerDied","Data":"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4"} Nov 02 23:32:47.879588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.879567 4494 scope.go:122] "RemoveContainer" containerID="c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4" Nov 02 23:32:47.881027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.881000 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:47.881109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.881073 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:47.881159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.881108 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:32:47.962243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:47.962202 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:48.111250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.111117 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:48.421522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.421493 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:48.421702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.421557 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:48.625898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.625866 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:48.794833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.794729 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:48.899048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.899012 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerStarted","Data":"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2"} Nov 02 23:32:48.987224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:48.986901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:49.147261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.144762 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:49.258287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.253585 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:32:49.258287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.253729 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:49.258287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.253773 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:49.421884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.421854 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 02 23:32:49.422076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:49.421922 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 02 23:32:50.254200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:50.253704 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:50.254200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:50.253769 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:50.801353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:50.801310 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:50.801859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:50.801828 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:50.802827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:50.802809 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:32:51.065244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.065134 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:51.255739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.255699 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:51.256446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.255794 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:51.256446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.255971 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:51.497199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.495025 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 02 23:32:51.505437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.505064 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.529033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.529001 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.529260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.529241 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh6v\" (UniqueName: \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.569249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.555587 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 02 23:32:51.570260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.560165 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f46b61\"" Nov 02 23:32:51.691211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.691069 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.691434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.691380 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh6v\" (UniqueName: \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.737122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.693141 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.801756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.801686 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh6v\" (UniqueName: \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\") pod \"payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:51.822204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:51.822146 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:52.259202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.253285 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 02 23:32:52.259202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.256317 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:52.259202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.256372 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:52.804565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.804538 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:32:52.968111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.968076 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" event={"ID":"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0","Type":"ContainerStarted","Data":"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754"} Nov 02 23:32:52.968472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.968451 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" event={"ID":"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0","Type":"ContainerStarted","Data":"94193efc083e066423593ec0660a19f5f32bf5a771073f5ae2035d5307576e1e"} Nov 02 23:32:52.969264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:52.969241 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:32:53.086897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:53.086756 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" podStartSLOduration=2.086735706 podStartE2EDuration="2.086735706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:53.085421238 +0000 UTC m=+2687.515291101" watchObservedRunningTime="2026-11-02 23:32:53.086735706 +0000 UTC m=+2687.516605569" Nov 02 23:32:53.251200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:53.250729 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:53.251200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:53.250804 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:53.816329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:53.816301 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:32:54.250393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:54.250354 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 02 23:32:54.250599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:54.250423 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 02 23:32:54.453381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:54.453340 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:55.254818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:55.254783 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:32:55.255400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:55.255260 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:32:55.255944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:55.255923 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:32:58.340201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:58.310145 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:58.598261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:58.592960 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:58.856340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:58.856250 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 02 23:32:58.999124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:58.999086 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 02 23:32:58.999526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:58.999459 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.092198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.073475 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-440c60-dockercfg-skjr5\"" Nov 02 23:32:59.092198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.073574 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-440c60\"" Nov 02 23:32:59.219936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.219893 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.220371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.220346 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2fj\" (UniqueName: \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.325091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.324087 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.325499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.325474 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2fj\" (UniqueName: \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.326036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.325045 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.451208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.444465 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2fj\" (UniqueName: \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\") pod \"payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.508739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.508450 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:32:59.563270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.563233 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:32:59.701277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.690021 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:32:59.997252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:32:59.992049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:00.206205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:00.191827 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:33:00.269585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:00.269486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:00.514579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:00.514542 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 02 23:33:00.521128 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:33:00.521093 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f1c709_ac51_46d8_923c_0b92234ca076.slice/crio-d1f740ad17907cdc0d341070c189c6e8bfb4b5f7e607bf7bb8c529e058884070 WatchSource:0}: Error finding container d1f740ad17907cdc0d341070c189c6e8bfb4b5f7e607bf7bb8c529e058884070: Status 404 returned error can't find the container with id d1f740ad17907cdc0d341070c189c6e8bfb4b5f7e607bf7bb8c529e058884070 Nov 02 23:33:00.837569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:00.837481 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:01.277415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:01.277382 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" event={"ID":"20f1c709-ac51-46d8-923c-0b92234ca076","Type":"ContainerStarted","Data":"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594"} Nov 02 23:33:01.277612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:01.277433 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" event={"ID":"20f1c709-ac51-46d8-923c-0b92234ca076","Type":"ContainerStarted","Data":"d1f740ad17907cdc0d341070c189c6e8bfb4b5f7e607bf7bb8c529e058884070"} Nov 02 23:33:01.277612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:01.277456 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:33:01.400198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:01.399682 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" podStartSLOduration=3.399663099 podStartE2EDuration="3.399663099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:01.398598631 +0000 UTC m=+2695.828468496" watchObservedRunningTime="2026-11-02 23:33:01.399663099 +0000 UTC m=+2695.829532962" Nov 02 23:33:04.992252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:04.992223 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 02 23:33:05.064591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.064555 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" podUID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" containerName="payload-processing" containerID="cri-o://ad92dde55ce24dd21c4b42e438620090c9430d3377f0d9a4b472b504935c253a" gracePeriod=25 Nov 02 23:33:05.208306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.208236 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" podUID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" containerName="payload-pre-processing" containerID="cri-o://c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270" gracePeriod=25 Nov 02 23:33:05.307404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.307367 4494 generic.go:356] "Generic (PLEG): container finished" podID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" containerID="ad92dde55ce24dd21c4b42e438620090c9430d3377f0d9a4b472b504935c253a" exitCode=0 Nov 02 23:33:05.307570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.307445 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" event={"ID":"e882f79f-d81d-4eb3-afb2-37070f62ddc5","Type":"ContainerDied","Data":"ad92dde55ce24dd21c4b42e438620090c9430d3377f0d9a4b472b504935c253a"} Nov 02 23:33:05.351187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.351144 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:33:05.404440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.404411 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\") pod \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " Nov 02 23:33:05.404607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.404526 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\" (UniqueName: \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\") pod \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\" (UID: \"e882f79f-d81d-4eb3-afb2-37070f62ddc5\") " Nov 02 23:33:05.404794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.404772 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume" pod "e882f79f-d81d-4eb3-afb2-37070f62ddc5" (UID: "e882f79f-d81d-4eb3-afb2-37070f62ddc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:05.405038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.405020 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e882f79f-d81d-4eb3-afb2-37070f62ddc5-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.406981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.406957 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w" pod "e882f79f-d81d-4eb3-afb2-37070f62ddc5" (UID: "e882f79f-d81d-4eb3-afb2-37070f62ddc5"). InnerVolumeSpecName "kube-api-access-bgr7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:05.467030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.467005 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:33:05.505843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.505775 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\" (UniqueName: \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\") pod \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " Nov 02 23:33:05.506054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.506032 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\") pod \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\" (UID: \"0e3c73df-f0d3-4f60-9fc7-10dd090adcce\") " Nov 02 23:33:05.506525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.506504 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgr7w\" (UniqueName: \"kubernetes.io/projected/e882f79f-d81d-4eb3-afb2-37070f62ddc5-kube-api-access-bgr7w\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.506525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.506513 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume" pod "0e3c73df-f0d3-4f60-9fc7-10dd090adcce" (UID: "0e3c73df-f0d3-4f60-9fc7-10dd090adcce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:05.507834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.507813 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj" pod "0e3c73df-f0d3-4f60-9fc7-10dd090adcce" (UID: "0e3c73df-f0d3-4f60-9fc7-10dd090adcce"). InnerVolumeSpecName "kube-api-access-wrjhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:05.607804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.607764 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.607804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.607804 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjhj\" (UniqueName: \"kubernetes.io/projected/0e3c73df-f0d3-4f60-9fc7-10dd090adcce-kube-api-access-wrjhj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.863114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:05.863021 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:06.316266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.316228 4494 generic.go:356] "Generic (PLEG): container finished" podID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" containerID="c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270" exitCode=0 Nov 02 23:33:06.316736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.316299 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" Nov 02 23:33:06.316736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.316303 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" event={"ID":"0e3c73df-f0d3-4f60-9fc7-10dd090adcce","Type":"ContainerDied","Data":"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270"} Nov 02 23:33:06.316736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.316381 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr" event={"ID":"0e3c73df-f0d3-4f60-9fc7-10dd090adcce","Type":"ContainerDied","Data":"c681425d2f0ad09885db7cfaba9e2444f242f016cb774ad3ecd79d2f0f0190f2"} Nov 02 23:33:06.316736 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.316405 4494 scope.go:122] "RemoveContainer" containerID="c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270" Nov 02 23:33:06.318055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.317892 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" event={"ID":"e882f79f-d81d-4eb3-afb2-37070f62ddc5","Type":"ContainerDied","Data":"16bd2c9e0a9f3e5a9b3a1d3d2a61a99ae48fb470beb48cefc009226f1821a6c3"} Nov 02 23:33:06.318055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.317897 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh" Nov 02 23:33:06.326870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.326849 4494 scope.go:122] "RemoveContainer" containerID="c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270" Nov 02 23:33:06.327150 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:33:06.327129 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270\": container with ID starting with c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270 not found: ID does not exist" containerID="c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270" Nov 02 23:33:06.327245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.327159 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270"} err="failed to get container status \"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270\": rpc error: code = NotFound desc = could not find container \"c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270\": container with ID starting with c7a9a92594421f15f450aa68e4bb8a757baeb007abaee0903b894fa409b23270 not found: ID does not exist" Nov 02 23:33:06.327245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.327202 4494 scope.go:122] "RemoveContainer" containerID="ad92dde55ce24dd21c4b42e438620090c9430d3377f0d9a4b472b504935c253a" Nov 02 23:33:06.485027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.484994 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:33:06.534937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.534908 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2279ad-6bfd64d8c5-srltr"] Nov 02 23:33:06.599187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.599092 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:33:06.604705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:06.604675 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2279ad-84c8c5c497-2h4nh"] Nov 02 23:33:07.703595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:07.703560 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" path="/var/lib/kubelet/pods/0e3c73df-f0d3-4f60-9fc7-10dd090adcce/volumes" Nov 02 23:33:07.704023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:07.704008 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" path="/var/lib/kubelet/pods/e882f79f-d81d-4eb3-afb2-37070f62ddc5/volumes" Nov 02 23:33:10.747859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:10.747824 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:10.892280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:10.892244 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:11.445913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:11.445872 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:33:12.288377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:12.288336 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 02 23:33:13.355236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:13.355136 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:14.150232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150197 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 02 23:33:14.150807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150793 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" containerName="payload-pre-processing" Nov 02 23:33:14.150875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150807 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" containerName="payload-pre-processing" Nov 02 23:33:14.150875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150854 4494 cpu_manager.go:414] "containerMap: removing container" podUID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" containerName="payload-processing" Nov 02 23:33:14.150875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150859 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" containerName="payload-processing" Nov 02 23:33:14.150993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150936 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e882f79f-d81d-4eb3-afb2-37070f62ddc5" containerName="payload-processing" Nov 02 23:33:14.150993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.150947 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3c73df-f0d3-4f60-9fc7-10dd090adcce" containerName="payload-pre-processing" Nov 02 23:33:14.156612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.156594 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.185774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.185740 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.185959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.185848 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.185959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.185885 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.185959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.185910 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.186120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.186005 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.186120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.186036 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9v8r\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.186120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.186081 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.186120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.186104 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.186341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.186208 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.210059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.210016 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-980b82-openshift-default-dockercfg-vwtdl\"" Nov 02 23:33:14.240032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.240003 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:14.265072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.265039 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 02 23:33:14.292804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.292763 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.293701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.293676 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9v8r\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.294353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.294321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.295411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.295392 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.296097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.296080 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.296738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.295343 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.296940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.296526 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.297104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.295953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.298598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.297906 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.298854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.298776 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.299436 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.299289 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.299882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.299837 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.300021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.299372 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.300282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.300265 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.361095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.356136 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.364091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.363901 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.366352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.366325 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.366458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.366329 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9v8r\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\") pod \"e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.456995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.456882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:14.471879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.471829 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:14.902672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.902643 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 02 23:33:14.903765 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:33:14.903741 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8426ee_2058_4c4a_b96d_c503316a01bf.slice/crio-a21406ea52f19c6a30515e88dcb4488e07059d93c183fc1d5979a338e15b547b WatchSource:0}: Error finding container a21406ea52f19c6a30515e88dcb4488e07059d93c183fc1d5979a338e15b547b: Status 404 returned error can't find the container with id a21406ea52f19c6a30515e88dcb4488e07059d93c183fc1d5979a338e15b547b Nov 02 23:33:14.906463 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.906430 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:33:14.906579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.906508 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:33:14.906579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:14.906555 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:33:15.400337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.400305 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerStarted","Data":"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc"} Nov 02 23:33:15.400802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.400359 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerStarted","Data":"a21406ea52f19c6a30515e88dcb4488e07059d93c183fc1d5979a338e15b547b"} Nov 02 23:33:15.474110 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.473242 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:15.479357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.479331 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:15.522768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.522713 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podStartSLOduration=1.522699356 podStartE2EDuration="1.522699356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:15.521516137 +0000 UTC m=+2709.951386022" watchObservedRunningTime="2026-11-02 23:33:15.522699356 +0000 UTC m=+2709.952569216" Nov 02 23:33:15.990474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:15.990408 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:16.407030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:16.406949 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:16.407995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:16.407976 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:33:17.559115 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:33:17.559072 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:33:18.299524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:18.299491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:18.463359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:18.463329 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:21.438954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:21.438913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:21.643107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:21.643071 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:22.102425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.102393 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 02 23:33:22.137264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.137234 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 02 23:33:22.137493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.137477 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.183167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.183131 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.183373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.183310 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vv5\" (UniqueName: \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.192550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.192528 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-980b82\"" Nov 02 23:33:22.243108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.243081 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-980b82-dockercfg-kcx4j\"" Nov 02 23:33:22.284094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.284062 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vv5\" (UniqueName: \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.284324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.284242 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.285136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.285113 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.346299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.346266 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vv5\" (UniqueName: \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\") pod \"payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.453901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.453862 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:22.862438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:22.862409 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 02 23:33:22.863016 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:33:22.862994 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a4b01_50be_41cb_9cff_2bf4ada7592d.slice/crio-8d28d99ede24448cbcbf66b7f13abfa735dfba55c856fadfc01c9dae4c1ee196 WatchSource:0}: Error finding container 8d28d99ede24448cbcbf66b7f13abfa735dfba55c856fadfc01c9dae4c1ee196: Status 404 returned error can't find the container with id 8d28d99ede24448cbcbf66b7f13abfa735dfba55c856fadfc01c9dae4c1ee196 Nov 02 23:33:23.009238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.009207 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:23.176145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.176116 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:23.370944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.370909 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:23.459596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.459485 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" event={"ID":"384a4b01-50be-41cb-9cff-2bf4ada7592d","Type":"ContainerStarted","Data":"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7"} Nov 02 23:33:23.459596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.459535 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" event={"ID":"384a4b01-50be-41cb-9cff-2bf4ada7592d","Type":"ContainerStarted","Data":"8d28d99ede24448cbcbf66b7f13abfa735dfba55c856fadfc01c9dae4c1ee196"} Nov 02 23:33:23.459596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.459589 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:23.578113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:23.578052 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" podStartSLOduration=2.57803366 podStartE2EDuration="2.57803366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:23.577124962 +0000 UTC m=+2718.006994823" watchObservedRunningTime="2026-11-02 23:33:23.57803366 +0000 UTC m=+2718.007903531" Nov 02 23:33:28.672940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:28.672902 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:33.642530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:33.642493 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:34.469432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:34.469402 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 02 23:33:36.851203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:36.851137 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:36.999460 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:36.999418 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:38.238040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.238006 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 02 23:33:38.242070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.242042 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.346427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.346394 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f46b61-serving-cert\"" Nov 02 23:33:38.350373 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.350352 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 02 23:33:38.366641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.366614 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.366842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.366715 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.366931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.366909 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.366989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.366969 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbcb\" (UniqueName: \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.433458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.433423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:38.468683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.468655 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.468894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.468704 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbcb\" (UniqueName: \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.468894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.468756 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.468894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.468838 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.469576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.469461 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.471327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.471306 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.471412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.471333 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.530803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.530698 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbcb\" (UniqueName: \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\") pod \"maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.556351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.556324 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:38.578511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.578479 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:38.736254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.735126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:38.935682 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:33:38.935651 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0b840d_1529_48aa_98ae_bff9989f9548.slice/crio-9a6b69a2c64502df3db8a2262c3c59331e67f7bd03cc7f1637009714d7836135 WatchSource:0}: Error finding container 9a6b69a2c64502df3db8a2262c3c59331e67f7bd03cc7f1637009714d7836135: Status 404 returned error can't find the container with id 9a6b69a2c64502df3db8a2262c3c59331e67f7bd03cc7f1637009714d7836135 Nov 02 23:33:38.937315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:38.937288 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 02 23:33:39.626038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:39.626007 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" event={"ID":"7d0b840d-1529-48aa-98ae-bff9989f9548","Type":"ContainerStarted","Data":"9a6b69a2c64502df3db8a2262c3c59331e67f7bd03cc7f1637009714d7836135"} Nov 02 23:33:40.634990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:40.634956 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" event={"ID":"7d0b840d-1529-48aa-98ae-bff9989f9548","Type":"ContainerStarted","Data":"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966"} Nov 02 23:33:40.635422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:40.635055 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:40.755727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:40.755665 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" podStartSLOduration=1.938100849 podStartE2EDuration="2.755651979s" totalImagesPullingTime="817.55113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:40.75333566 +0000 UTC m=+2735.183205520" watchObservedRunningTime="2026-11-02 23:33:40.755651979 +0000 UTC m=+2735.185521844" Nov 02 23:33:43.445113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:43.445081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:43.599891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:43.599861 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:43.747647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:43.747542 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:46.647807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:46.647779 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 02 23:33:46.756262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:46.756158 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:46.900532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:46.900444 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:47.561945 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:33:47.561844 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:33:48.861448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:48.861416 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:51.537680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:51.537646 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:33:52.430774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:52.430736 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:52.921735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:52.921705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:33:55.989580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:33:55.989511 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:00.966395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:00.966354 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:01.902339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:01.902303 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:34:01.903363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:01.902383 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:34:02.853935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.853895 4494 generic.go:356] "Generic (PLEG): container finished" podID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerID="c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2" exitCode=0 Nov 02 23:34:02.854480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.853969 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerDied","Data":"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2"} Nov 02 23:34:02.854480 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.854406 4494 scope.go:122] "RemoveContainer" containerID="c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2" Nov 02 23:34:02.855763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.855739 4494 generic.go:356] "Generic (PLEG): container finished" podID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerID="39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893" exitCode=0 Nov 02 23:34:02.855895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.855818 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerDied","Data":"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893"} Nov 02 23:34:02.855998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.855975 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:02.856077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.856060 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:02.856144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.856115 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:02.856235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.856122 4494 scope.go:122] "RemoveContainer" containerID="39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893" Nov 02 23:34:02.857643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.857614 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:02.857726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.857689 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:02.857769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:02.857734 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:03.888570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:03.888534 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerStarted","Data":"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c"} Nov 02 23:34:03.890707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:03.890678 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerStarted","Data":"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f"} Nov 02 23:34:04.383499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.383396 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:34:04.384649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.384626 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 23:34:04.384714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.384694 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 23:34:04.473196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.472978 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:34:04.505851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.505826 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:34:04.506087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.505864 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 23:34:04.506228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:04.506205 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 23:34:05.386253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:05.385516 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 23:34:05.386253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:05.385582 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 23:34:05.506328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:05.506295 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 23:34:05.506500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:05.506362 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 23:34:05.987298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:05.985606 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:06.383570 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:06.383474 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 23:34:06.383737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:06.383557 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 23:34:06.505797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:06.505760 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 23:34:06.506239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:06.505827 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 23:34:07.384688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:07.384657 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 23:34:07.386080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:07.385509 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 23:34:07.506405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:07.506368 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 23:34:07.506813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:07.506418 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 23:34:07.565617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:07.565315 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:08.383894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:08.383861 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Nov 02 23:34:08.384081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:08.383918 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Nov 02 23:34:08.506338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:08.505750 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 23:34:08.506338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:08.505821 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 23:34:09.388833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:09.388797 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:34:09.389748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:09.389729 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:34:09.510577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:09.510536 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:34:09.510960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:09.510857 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:34:09.511743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:09.511726 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 02 23:34:11.055640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:11.055605 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:11.201949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:11.201905 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:12.944589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:12.944552 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:15.934240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:15.934190 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:34:16.158806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:16.158774 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:17.591218 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:34:17.591156 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:34:19.149501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:19.149455 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:23.023310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:23.023261 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:23.207335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:23.207299 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:23.351649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:23.351563 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:25.051291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:25.051254 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w"] Nov 02 23:34:25.103681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:25.103650 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5eee018-4mp8w"] Nov 02 23:34:25.696579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:25.696537 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc44259-a2bb-43a7-a475-ad2c41611185" path="/var/lib/kubelet/pods/3dc44259-a2bb-43a7-a475-ad2c41611185/volumes" Nov 02 23:34:26.074410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.072733 4494 generic.go:356] "Generic (PLEG): container finished" podID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerID="ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e" exitCode=0 Nov 02 23:34:26.074410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.072864 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerDied","Data":"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e"} Nov 02 23:34:26.074410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.073438 4494 scope.go:122] "RemoveContainer" containerID="ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e" Nov 02 23:34:26.078077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.077222 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:26.078077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.077320 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:26.078077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:26.077368 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:27.082597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:27.082561 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerStarted","Data":"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3"} Nov 02 23:34:27.788050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:27.788017 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:34:27.789284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:27.789251 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 23:34:27.789439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:27.789312 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 23:34:28.379372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:28.379342 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:28.788114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:28.788074 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 23:34:28.788337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:28.788130 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 23:34:29.788576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:29.788542 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 23:34:29.788989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:29.788612 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 23:34:30.787939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:30.787901 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 23:34:30.788152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:30.787963 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 23:34:31.787911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:31.787877 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 23:34:31.788431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:31.787931 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 23:34:31.870489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:31.870458 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:34:32.800427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:32.800379 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:34:32.804862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:32.804835 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 02 23:34:33.442849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:33.442808 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:34.971121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:34.971076 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:38.748517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:38.748466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:38.975123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:38.975089 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:40.153243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.153211 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:40.670088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.670054 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:34:40.673100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.673084 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.722301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.722245 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.723252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.723164 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.723552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.723526 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.723711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.723695 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.723882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.723867 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.724041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.724027 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.724276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.724259 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsht\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.724425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.724409 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.724534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.724518 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.735438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.735405 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a5d6d8-openshift-default-dockercfg-2rzcv\"" Nov 02 23:34:40.808196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.804273 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828207 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828282 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828322 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828379 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828763 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828840 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsht\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828884 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.828907 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.829866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.829259 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.855271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.841879 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.855271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.849369 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.855271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.850200 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.855271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.853385 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.855666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.855327 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.859541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.859484 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.865214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.864892 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.906861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.887994 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:34:40.932270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.929987 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.981333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.978898 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsht\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\") pod \"e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:40.985505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:40.985482 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:41.011064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:41.011033 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:41.474925 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:34:41.474895 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510b384f_58fa_415b_be77_feb8e7eb519c.slice/crio-e19bd7cffd032baa34b1325fabeadd079a10aa6d63eb79a0d06080ef78064acb WatchSource:0}: Error finding container e19bd7cffd032baa34b1325fabeadd079a10aa6d63eb79a0d06080ef78064acb: Status 404 returned error can't find the container with id e19bd7cffd032baa34b1325fabeadd079a10aa6d63eb79a0d06080ef78064acb Nov 02 23:34:41.475369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:41.475212 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:34:41.477532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:41.477506 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:41.477604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:41.477565 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:41.477604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:41.477593 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:42.187725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:42.187686 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerStarted","Data":"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf"} Nov 02 23:34:42.187921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:42.187750 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerStarted","Data":"e19bd7cffd032baa34b1325fabeadd079a10aa6d63eb79a0d06080ef78064acb"} Nov 02 23:34:42.313464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:42.313399 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podStartSLOduration=2.313378963 podStartE2EDuration="2.313378963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:42.30853846 +0000 UTC m=+2796.738408323" watchObservedRunningTime="2026-11-02 23:34:42.313378963 +0000 UTC m=+2796.743248824" Nov 02 23:34:42.985863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:42.985831 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:42.991388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:42.991359 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:43.047643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.047610 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j"] Nov 02 23:34:43.097821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.097794 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80628867-dg62j"] Nov 02 23:34:43.195100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.195067 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:43.196006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.195983 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:34:43.693869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.693835 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca28050-3f6b-4cab-a72f-c8916c07a8ce" path="/var/lib/kubelet/pods/2ca28050-3f6b-4cab-a72f-c8916c07a8ce/volumes" Nov 02 23:34:43.876877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:43.876845 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:45.389522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:45.389484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:45.558389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:45.558354 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:45.705243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:45.705196 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:47.559720 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:34:47.559686 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:34:48.618681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:48.618638 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:48.821204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:48.807131 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:49.223250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.215805 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:34:49.270989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.268349 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:34:49.270989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.268479 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.331141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.331101 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:49.335798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.335776 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a5d6d8-dockercfg-cwmjz\"" Nov 02 23:34:49.337542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.337519 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.337834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.337819 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgtl\" (UniqueName: \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.338365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.338329 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a5d6d8\"" Nov 02 23:34:49.442999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.441266 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgtl\" (UniqueName: \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.442999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.441540 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.442999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.442658 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.504394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.504320 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgtl\" (UniqueName: \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\") pod \"payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:49.633854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:49.633817 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:50.020085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.020053 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:34:50.020588 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:34:50.020558 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbbbcd4_fd16_4f1d_b699_f00d6a3c79c4.slice/crio-11c863cc4d6461c2a3e11e08a0f9fe33c03ec560a5c7b693e39b12c63ec5661b WatchSource:0}: Error finding container 11c863cc4d6461c2a3e11e08a0f9fe33c03ec560a5c7b693e39b12c63ec5661b: Status 404 returned error can't find the container with id 11c863cc4d6461c2a3e11e08a0f9fe33c03ec560a5c7b693e39b12c63ec5661b Nov 02 23:34:50.250592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.250558 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:50.330971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.330876 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" event={"ID":"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4","Type":"ContainerStarted","Data":"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7"} Nov 02 23:34:50.330971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.330934 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" event={"ID":"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4","Type":"ContainerStarted","Data":"11c863cc4d6461c2a3e11e08a0f9fe33c03ec560a5c7b693e39b12c63ec5661b"} Nov 02 23:34:50.330971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.330959 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:34:50.448562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.448509 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" podStartSLOduration=1.448496609 podStartE2EDuration="1.448496609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:50.447120719 +0000 UTC m=+2804.876990610" watchObservedRunningTime="2026-11-02 23:34:50.448496609 +0000 UTC m=+2804.878366470" Nov 02 23:34:50.746147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:50.746107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:51.350424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.350393 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 02 23:34:51.354962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.354946 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384434 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384528 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384566 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384602 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384691 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384720 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384789 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9rc\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384815 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.385955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.384863 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.443347 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.498906 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.518379 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1f0009-openshift-default-dockercfg-8m9fq\"" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534525 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534580 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534614 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534650 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534727 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534756 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534812 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9rc\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534839 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.534885 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.536084 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.549012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.545242 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.567459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.567425 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.568575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.568546 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.572191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.568621 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.623451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.622608 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9rc\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.623451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.623326 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.703475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.703438 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.705762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.705479 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\") pod \"e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:51.984568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:51.984538 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:52.021391 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:52.021342 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:52.462587 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:34:52.462560 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d840d5_3a79_4ef6_8214_0bce2361db7b.slice/crio-f131b681f100487294a4b5f78d8af9ebd9908d7f7976b0518509b5e2842ef754 WatchSource:0}: Error finding container f131b681f100487294a4b5f78d8af9ebd9908d7f7976b0518509b5e2842ef754: Status 404 returned error can't find the container with id f131b681f100487294a4b5f78d8af9ebd9908d7f7976b0518509b5e2842ef754 Nov 02 23:34:52.463097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:52.463075 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 02 23:34:52.465381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:52.465347 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:52.465486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:52.465410 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:52.465486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:52.465443 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:34:53.354399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:53.354362 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerStarted","Data":"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965"} Nov 02 23:34:53.354855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:53.354419 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerStarted","Data":"f131b681f100487294a4b5f78d8af9ebd9908d7f7976b0518509b5e2842ef754"} Nov 02 23:34:53.578801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:53.578755 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podStartSLOduration=2.578739816 podStartE2EDuration="2.578739816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:53.525820265 +0000 UTC m=+2807.955690150" watchObservedRunningTime="2026-11-02 23:34:53.578739816 +0000 UTC m=+2808.008609676" Nov 02 23:34:54.008208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.003972 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:54.117572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.117394 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:54.361674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.361597 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:54.362594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.362574 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:34:54.726733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.726703 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:34:54.726932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.726899 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" containerID="cri-o://5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7" gracePeriod=30 Nov 02 23:34:54.981728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:54.981632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:56.270663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:56.270631 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:56.413609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:56.413576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:56.568007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:56.567926 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:56.744635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:56.744602 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:57.334197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:57.311992 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:57.824284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:57.824247 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:34:59.729980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:59.729947 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:34:59.730437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:59.729998 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:34:59.989410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:34:59.989345 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:35:00.085387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085355 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085431 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085504 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085595 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085645 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085671 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.085732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085676 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:00.085732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085720 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.086016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085783 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.086016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085855 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\") pod \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\" (UID: \"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d\") " Nov 02 23:35:00.086016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085872 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:00.086016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.085890 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:00.086274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086044 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:00.086369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086346 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:00.086809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086786 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.086931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086813 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.086931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086829 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.086931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086843 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.086931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.086859 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.088067 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.088042 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "kube-api-access-ncz99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:00.088217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.088069 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:00.088217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.088156 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:00.088217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.088190 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy" pod "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" (UID: "0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:00.100252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.100230 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b"] Nov 02 23:35:00.103996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.103976 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e6fa27-q275b"] Nov 02 23:35:00.188344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.188310 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncz99\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-kube-api-access-ncz99\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.188344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.188340 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.188344 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.188350 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.188591 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.188360 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:00.455432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.455388 4494 generic.go:356] "Generic (PLEG): container finished" podID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerID="5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7" exitCode=0 Nov 02 23:35:00.455644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.455469 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" Nov 02 23:35:00.455644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.455517 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" event={"ID":"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d","Type":"ContainerDied","Data":"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7"} Nov 02 23:35:00.455644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.455592 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg" event={"ID":"0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d","Type":"ContainerDied","Data":"65c965fa0c437efa2a1de154d290e3a317082a82f33afc772ae31e0583dd8108"} Nov 02 23:35:00.455644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.455630 4494 scope.go:122] "RemoveContainer" containerID="5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7" Nov 02 23:35:00.467441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.467417 4494 scope.go:122] "RemoveContainer" containerID="5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7" Nov 02 23:35:00.467743 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:00.467715 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7\": container with ID starting with 5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7 not found: ID does not exist" containerID="5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7" Nov 02 23:35:00.467828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.467757 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7"} err="failed to get container status \"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7\": rpc error: code = NotFound desc = could not find container \"5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7\": container with ID starting with 5cf3534b66513fe97f913928530d11b689553e290ea0100321be0d0a3f1da1e7 not found: ID does not exist" Nov 02 23:35:00.549641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.549611 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:00.627846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.627818 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:35:00.634858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.634828 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-zwbpg"] Nov 02 23:35:00.725249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.722887 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:00.960651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.960621 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 02 23:35:00.961353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.961337 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" Nov 02 23:35:00.961422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.961354 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" Nov 02 23:35:00.961473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.961464 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" containerName="istio-proxy" Nov 02 23:35:00.966744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:00.966724 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.024575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.023559 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1f0009-dockercfg-ncxnl\"" Nov 02 23:35:01.024575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.023570 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1f0009\"" Nov 02 23:35:01.024575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.024429 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 02 23:35:01.059720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.059218 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nv7\" (UniqueName: \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.059720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.059285 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.118002 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.117886 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 02 23:35:01.168377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.168337 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nv7\" (UniqueName: \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.168672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.168649 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.190198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.169832 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.213397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.213361 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 02 23:35:01.217587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.217561 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.252702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.252669 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nv7\" (UniqueName: \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\") pod \"payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.269840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.269769 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.270017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.269900 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kgk\" (UniqueName: \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.283003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.282929 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:01.344065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.344018 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:35:01.371387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.371350 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kgk\" (UniqueName: \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.371643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.371621 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.372586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.372562 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.433522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.433494 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kgk\" (UniqueName: \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\") pod \"payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.560921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.560852 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:01.664213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.664095 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 02 23:35:01.664367 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:01.664241 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6874d4_1de4_4de2_9aae_f4cab811cac2.slice/crio-dbe56bf4b06553f3a69281521f0ac1a88ae90cb9876a46b09227bcde9641c416 WatchSource:0}: Error finding container dbe56bf4b06553f3a69281521f0ac1a88ae90cb9876a46b09227bcde9641c416: Status 404 returned error can't find the container with id dbe56bf4b06553f3a69281521f0ac1a88ae90cb9876a46b09227bcde9641c416 Nov 02 23:35:01.697697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.697665 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c" path="/var/lib/kubelet/pods/02d7a4ee-6f3f-4cee-bbcf-87b831bdef8c/volumes" Nov 02 23:35:01.698161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.698143 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d" path="/var/lib/kubelet/pods/0fe9f9cf-3e4c-4712-95f7-a2cf25e3804d/volumes" Nov 02 23:35:01.953087 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:01.953056 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fd866e_2b8c_460b_b254_6fdf8f49a217.slice/crio-070eefac25f299bcf03868e797a0474ae8aa08a2db6f375522bb02dbde242346 WatchSource:0}: Error finding container 070eefac25f299bcf03868e797a0474ae8aa08a2db6f375522bb02dbde242346: Status 404 returned error can't find the container with id 070eefac25f299bcf03868e797a0474ae8aa08a2db6f375522bb02dbde242346 Nov 02 23:35:01.955040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:01.955013 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 02 23:35:02.472807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.472767 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" event={"ID":"39fd866e-2b8c-460b-b254-6fdf8f49a217","Type":"ContainerStarted","Data":"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12"} Nov 02 23:35:02.473350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.472835 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:02.473350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.472861 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" event={"ID":"39fd866e-2b8c-460b-b254-6fdf8f49a217","Type":"ContainerStarted","Data":"070eefac25f299bcf03868e797a0474ae8aa08a2db6f375522bb02dbde242346"} Nov 02 23:35:02.474469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.474445 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" event={"ID":"ec6874d4-1de4-4de2-9aae-f4cab811cac2","Type":"ContainerStarted","Data":"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e"} Nov 02 23:35:02.474622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.474481 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" event={"ID":"ec6874d4-1de4-4de2-9aae-f4cab811cac2","Type":"ContainerStarted","Data":"dbe56bf4b06553f3a69281521f0ac1a88ae90cb9876a46b09227bcde9641c416"} Nov 02 23:35:02.474622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.474529 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:02.590484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.590437 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" podStartSLOduration=1.590422646 podStartE2EDuration="1.590422646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:02.590060317 +0000 UTC m=+2817.019930177" watchObservedRunningTime="2026-11-02 23:35:02.590422646 +0000 UTC m=+2817.020292504" Nov 02 23:35:02.621442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.621411 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:02.724774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.724157 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" podStartSLOduration=2.724134905 podStartE2EDuration="2.724134905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:02.720361277 +0000 UTC m=+2817.150231162" watchObservedRunningTime="2026-11-02 23:35:02.724134905 +0000 UTC m=+2817.154004787" Nov 02 23:35:02.801303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:02.801255 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:03.000282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:03.000164 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:04.329910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:04.329880 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:35:05.995739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:05.995708 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:06.230401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.230362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:06.403927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.403846 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 02 23:35:06.408602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.408583 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.510903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.510110 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvks6\" (UniqueName: \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.510903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.510239 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.544275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.512685 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-915ccb\"" Nov 02 23:35:06.544275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.512851 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-915ccb-dockercfg-f2qd8\"" Nov 02 23:35:06.712315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.646254 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvks6\" (UniqueName: \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.712315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.646358 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.721428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.713468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:06.721428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.716529 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 02 23:35:06.775433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:06.775396 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvks6\" (UniqueName: \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\") pod \"payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:07.026238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:07.026131 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:07.463637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:07.463601 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 02 23:35:07.467507 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:07.467480 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515b5a7c_2f3f_4696_bc30_97d6b9c9d4df.slice/crio-ce4aae2eeb4b61c2810f892457af96df515e2d2f40a2d6761e69fcd224542d20 WatchSource:0}: Error finding container ce4aae2eeb4b61c2810f892457af96df515e2d2f40a2d6761e69fcd224542d20: Status 404 returned error can't find the container with id ce4aae2eeb4b61c2810f892457af96df515e2d2f40a2d6761e69fcd224542d20 Nov 02 23:35:07.755471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:07.755374 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" event={"ID":"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df","Type":"ContainerStarted","Data":"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5"} Nov 02 23:35:07.755701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:07.755663 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" event={"ID":"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df","Type":"ContainerStarted","Data":"ce4aae2eeb4b61c2810f892457af96df515e2d2f40a2d6761e69fcd224542d20"} Nov 02 23:35:07.755839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:07.755707 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:08.372871 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:08.372828 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:11.450196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:11.449646 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:13.485221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:13.485186 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 02 23:35:13.485806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:13.485269 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 02 23:35:13.653151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:13.653090 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" podStartSLOduration=7.653075395 podStartE2EDuration="7.653075395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:07.884476262 +0000 UTC m=+2822.314346126" watchObservedRunningTime="2026-11-02 23:35:13.653075395 +0000 UTC m=+2828.082945286" Nov 02 23:35:13.781747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:13.781621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:15.855055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.855025 4494 generic.go:356] "Generic (PLEG): container finished" podID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerID="aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc" exitCode=0 Nov 02 23:35:15.855535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.855094 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerDied","Data":"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc"} Nov 02 23:35:15.855535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.855495 4494 scope.go:122] "RemoveContainer" containerID="aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc" Nov 02 23:35:15.857150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.857122 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:15.857249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.857230 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:15.857299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:15.857277 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:16.407661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:16.407626 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:35:16.863825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:16.863793 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerStarted","Data":"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933"} Nov 02 23:35:17.472303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:17.472273 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:35:17.473275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:17.473247 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 02 23:35:17.473360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:17.473304 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 02 23:35:17.591785 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:17.591704 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:35:18.473139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:18.473103 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 02 23:35:18.473607 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:18.473187 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 02 23:35:18.766966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:18.766877 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 02 23:35:18.972958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:18.972926 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:19.473247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:19.473206 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 02 23:35:19.473735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:19.473275 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 02 23:35:20.473374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:20.473336 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 02 23:35:20.473841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:20.473428 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 02 23:35:21.473186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:21.473142 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 02 23:35:21.473377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:21.473216 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 02 23:35:22.476803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:22.476772 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:35:22.477414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:22.477392 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:35:22.477965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:22.477946 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 02 23:35:23.200601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.200570 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:35:23.200820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.200785 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" podUID="2825a71f-3473-4646-878a-2af4f3c27c22" containerName="maas-api" containerID="cri-o://6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d" gracePeriod=30 Nov 02 23:35:23.310714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.310679 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:23.537393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.537362 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:35:23.654197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.644132 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\" (UniqueName: \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\") pod \"2825a71f-3473-4646-878a-2af4f3c27c22\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " Nov 02 23:35:23.654197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.644273 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") pod \"2825a71f-3473-4646-878a-2af4f3c27c22\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " Nov 02 23:35:23.654197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.644354 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\") pod \"2825a71f-3473-4646-878a-2af4f3c27c22\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " Nov 02 23:35:23.654197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.644664 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\") pod \"2825a71f-3473-4646-878a-2af4f3c27c22\" (UID: \"2825a71f-3473-4646-878a-2af4f3c27c22\") " Nov 02 23:35:23.654197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.646069 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca" pod "2825a71f-3473-4646-878a-2af4f3c27c22" (UID: "2825a71f-3473-4646-878a-2af4f3c27c22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:23.657873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.657762 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f" pod "2825a71f-3473-4646-878a-2af4f3c27c22" (UID: "2825a71f-3473-4646-878a-2af4f3c27c22"). InnerVolumeSpecName "kube-api-access-9sr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:23.658536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.658496 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs" pod "2825a71f-3473-4646-878a-2af4f3c27c22" (UID: "2825a71f-3473-4646-878a-2af4f3c27c22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:23.680210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.678356 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls" pod "2825a71f-3473-4646-878a-2af4f3c27c22" (UID: "2825a71f-3473-4646-878a-2af4f3c27c22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:23.728758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.728721 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:35:23.729005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.728974 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" podUID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" containerName="payload-pre-processing" containerID="cri-o://ff2c5e05e03d2091e9c85778804a5ba045f86ff21bf3ae4323528f0ca4c7462c" gracePeriod=30 Nov 02 23:35:23.746374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.746352 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2825a71f-3473-4646-878a-2af4f3c27c22-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:23.746374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.746374 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr2f\" (UniqueName: \"kubernetes.io/projected/2825a71f-3473-4646-878a-2af4f3c27c22-kube-api-access-9sr2f\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:23.746562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.746384 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:23.746562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.746394 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2825a71f-3473-4646-878a-2af4f3c27c22-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:23.826644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.826553 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:23.927121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.927067 4494 generic.go:356] "Generic (PLEG): container finished" podID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" containerID="ff2c5e05e03d2091e9c85778804a5ba045f86ff21bf3ae4323528f0ca4c7462c" exitCode=0 Nov 02 23:35:23.927371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.927343 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" event={"ID":"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5","Type":"ContainerDied","Data":"ff2c5e05e03d2091e9c85778804a5ba045f86ff21bf3ae4323528f0ca4c7462c"} Nov 02 23:35:23.929342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.929315 4494 generic.go:356] "Generic (PLEG): container finished" podID="2825a71f-3473-4646-878a-2af4f3c27c22" containerID="6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d" exitCode=0 Nov 02 23:35:23.929505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.929377 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" Nov 02 23:35:23.929505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.929400 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" event={"ID":"2825a71f-3473-4646-878a-2af4f3c27c22","Type":"ContainerDied","Data":"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d"} Nov 02 23:35:23.929505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.929487 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9" event={"ID":"2825a71f-3473-4646-878a-2af4f3c27c22","Type":"ContainerDied","Data":"f87666b7d9fcf21faabd3b688c1ab4b14401078ac2f3f337fc65be8c13ea7ed1"} Nov 02 23:35:23.929679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.929547 4494 scope.go:122] "RemoveContainer" containerID="6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d" Nov 02 23:35:23.943284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.943259 4494 scope.go:122] "RemoveContainer" containerID="6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d" Nov 02 23:35:23.943553 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:23.943532 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d\": container with ID starting with 6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d not found: ID does not exist" containerID="6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d" Nov 02 23:35:23.943643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.943560 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d"} err="failed to get container status \"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d\": rpc error: code = NotFound desc = could not find container \"6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d\": container with ID starting with 6b5cd89b3aec2eff3284329e7689a8c96a9e8ea171b98120f2e0e8e347921e4d not found: ID does not exist" Nov 02 23:35:23.977354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.977329 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:35:23.981361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:23.981338 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:24.050973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.050945 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\" (UniqueName: \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\") pod \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " Nov 02 23:35:24.051208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.051052 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\") pod \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\" (UID: \"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5\") " Nov 02 23:35:24.051500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.051472 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume" pod "4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" (UID: "4fdb853b-39fd-481c-b7a6-4ef14e32a5c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:24.053382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.053360 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl" pod "4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" (UID: "4fdb853b-39fd-481c-b7a6-4ef14e32a5c5"). InnerVolumeSpecName "kube-api-access-j4djl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:24.101091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.098764 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:35:24.105656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.105632 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-818fe8-77bcd9fff6-jj6r9"] Nov 02 23:35:24.152995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.152912 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:24.152995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.152948 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4djl\" (UniqueName: \"kubernetes.io/projected/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5-kube-api-access-j4djl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:24.945808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.945769 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" Nov 02 23:35:24.945808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.945790 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m" event={"ID":"4fdb853b-39fd-481c-b7a6-4ef14e32a5c5","Type":"ContainerDied","Data":"690b9a5a08e4351696f11d55e1b9a74362907840176df4515c84ec886acd2c74"} Nov 02 23:35:24.946407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:24.945850 4494 scope.go:122] "RemoveContainer" containerID="ff2c5e05e03d2091e9c85778804a5ba045f86ff21bf3ae4323528f0ca4c7462c" Nov 02 23:35:25.116754 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:25.116727 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:35:25.120768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:25.120743 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-818fe8-9dfdcc6ff-fd86m"] Nov 02 23:35:25.150129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:25.150099 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:25.695399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:25.695357 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2825a71f-3473-4646-878a-2af4f3c27c22" path="/var/lib/kubelet/pods/2825a71f-3473-4646-878a-2af4f3c27c22/volumes" Nov 02 23:35:25.695795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:25.695779 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" path="/var/lib/kubelet/pods/4fdb853b-39fd-481c-b7a6-4ef14e32a5c5/volumes" Nov 02 23:35:27.322336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:27.322284 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:27.468291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:27.468243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:28.906545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:28.906510 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:29.446303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:29.446238 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:29.446832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:29.446815 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:35:29.447016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:29.446999 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" containerID="cri-o://d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937" gracePeriod=30 Nov 02 23:35:29.741644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:29.741564 4494 patch_prober.go:29] interesting pod/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:35:29.741644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:29.741613 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:35:30.566806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:30.566766 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:34.709827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.709753 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:35:34.803019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.802982 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803054 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803226 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803315 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803336 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.803385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803353 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803411 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803493 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803538 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803604 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\") pod \"1113f649-222b-43fe-b68d-d3138cf68354\" (UID: \"1113f649-222b-43fe-b68d-d3138cf68354\") " Nov 02 23:35:34.803649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803618 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.803817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803649 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:34.803899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803843 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.803959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.803896 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.804623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.804521 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.804623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.804551 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1113f649-222b-43fe-b68d-d3138cf68354-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.804623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.804567 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.804623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.804583 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.804623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.804598 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.805763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.805730 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:34.805887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.805852 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:34.805972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.805949 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "kube-api-access-2ng6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:34.806082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.806064 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo" pod "1113f649-222b-43fe-b68d-d3138cf68354" (UID: "1113f649-222b-43fe-b68d-d3138cf68354"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:34.905741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.905709 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ng6f\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-kube-api-access-2ng6f\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.905741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.905739 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1113f649-222b-43fe-b68d-d3138cf68354-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.905955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.905750 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1113f649-222b-43fe-b68d-d3138cf68354-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:34.905955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:34.905759 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1113f649-222b-43fe-b68d-d3138cf68354-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:35:35.030627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.030543 4494 generic.go:356] "Generic (PLEG): container finished" podID="1113f649-222b-43fe-b68d-d3138cf68354" containerID="d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937" exitCode=0 Nov 02 23:35:35.030627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.030587 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" Nov 02 23:35:35.030868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.030623 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" event={"ID":"1113f649-222b-43fe-b68d-d3138cf68354","Type":"ContainerDied","Data":"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937"} Nov 02 23:35:35.030868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.030675 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm" event={"ID":"1113f649-222b-43fe-b68d-d3138cf68354","Type":"ContainerDied","Data":"2e2da3cffdbfc4b5d6d70a4f37e061116ada345bbccff96d570b37cf125fa01e"} Nov 02 23:35:35.030868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.030698 4494 scope.go:122] "RemoveContainer" containerID="d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937" Nov 02 23:35:35.041881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.041854 4494 scope.go:122] "RemoveContainer" containerID="d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937" Nov 02 23:35:35.042125 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:35.042110 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937\": container with ID starting with d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937 not found: ID does not exist" containerID="d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937" Nov 02 23:35:35.042192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.042134 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937"} err="failed to get container status \"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937\": rpc error: code = NotFound desc = could not find container \"d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937\": container with ID starting with d0975292f3ea1a81bba2f0b36dfa589c8c29b1960ffe5c966c10b03e02883937 not found: ID does not exist" Nov 02 23:35:35.153130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.153096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:35.202873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.202838 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:35:35.216192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.211117 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-818fe8-openshift-default-f54d68bf9-xvvmm"] Nov 02 23:35:35.694225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.694189 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1113f649-222b-43fe-b68d-d3138cf68354" path="/var/lib/kubelet/pods/1113f649-222b-43fe-b68d-d3138cf68354/volumes" Nov 02 23:35:35.884278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:35.884237 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:36.036954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.036839 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:36.444517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.444482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:36.995012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.994978 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 02 23:35:36.995626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995612 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" containerName="payload-pre-processing" Nov 02 23:35:36.995675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995628 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" containerName="payload-pre-processing" Nov 02 23:35:36.995675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995665 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2825a71f-3473-4646-878a-2af4f3c27c22" containerName="maas-api" Nov 02 23:35:36.995675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995670 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2825a71f-3473-4646-878a-2af4f3c27c22" containerName="maas-api" Nov 02 23:35:36.995788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995686 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" Nov 02 23:35:36.995788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995691 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" Nov 02 23:35:36.995868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995812 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2825a71f-3473-4646-878a-2af4f3c27c22" containerName="maas-api" Nov 02 23:35:36.995868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995823 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1113f649-222b-43fe-b68d-d3138cf68354" containerName="istio-proxy" Nov 02 23:35:36.995868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.995833 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fdb853b-39fd-481c-b7a6-4ef14e32a5c5" containerName="payload-pre-processing" Nov 02 23:35:36.999534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:36.999516 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.037578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.037547 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.037761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.037626 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.037827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.037781 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.037923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.037892 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.037984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.037951 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.038113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.038049 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.038321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.038254 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvjh\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.038321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.038303 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.038444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.038334 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.073796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.073765 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 02 23:35:37.098044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.098018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:37.109596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.109573 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b8d3cb-openshift-default-dockercfg-4rl92\"" Nov 02 23:35:37.139332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139302 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139395 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139448 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139481 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139529 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139632 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvjh\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139672 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139722 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139706 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139745 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139771 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139804 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.139940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.139842 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.140103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.140070 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.140443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.140398 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.142028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.142001 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.142425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.142404 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.201211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.201162 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.201351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.201238 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvjh\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\") pod \"e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.248181 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.247848 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:37.318601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.318275 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:37.825531 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:37.825493 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48face08_0636_4c00_b7e8_9e8de6ce7983.slice/crio-5eacfcaedb291fd8cb6c6f38f4e362cf6ae5138504acc51344b12ee2202b4440 WatchSource:0}: Error finding container 5eacfcaedb291fd8cb6c6f38f4e362cf6ae5138504acc51344b12ee2202b4440: Status 404 returned error can't find the container with id 5eacfcaedb291fd8cb6c6f38f4e362cf6ae5138504acc51344b12ee2202b4440 Nov 02 23:35:37.826129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.826048 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 02 23:35:37.828322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.828292 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:37.828401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.828358 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:37.828401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:37.828386 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:35:38.059560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:38.059525 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerStarted","Data":"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945"} Nov 02 23:35:38.060068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:38.059580 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerStarted","Data":"5eacfcaedb291fd8cb6c6f38f4e362cf6ae5138504acc51344b12ee2202b4440"} Nov 02 23:35:38.183616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:38.183566 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podStartSLOduration=2.183550606 podStartE2EDuration="2.183550606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:38.180582009 +0000 UTC m=+2852.610451871" watchObservedRunningTime="2026-11-02 23:35:38.183550606 +0000 UTC m=+2852.613420516" Nov 02 23:35:38.323624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:38.323589 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:38.342044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:38.342002 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:39.067424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:39.067388 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:39.068517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:39.068498 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:35:40.984770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:40.984730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:41.188716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:41.188682 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:41.332308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:41.332200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:41.539546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:41.539506 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:41.684873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:41.684842 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:43.165412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:43.165377 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:35:44.306393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:44.306351 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:44.453059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:44.453023 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:44.606141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:44.606042 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:44.814239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:44.811479 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:45.065668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.065636 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l"] Nov 02 23:35:45.116304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.116277 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d616dfa4-snx2l"] Nov 02 23:35:45.186545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.180196 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 02 23:35:45.220198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.213553 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.294201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.271013 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 02 23:35:45.297189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.295683 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglp4\" (UniqueName: \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.297189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.295820 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.297189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.296961 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b8d3cb-dockercfg-72bqw\"" Nov 02 23:35:45.297189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.297067 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b8d3cb\"" Nov 02 23:35:45.397130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.396993 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglp4\" (UniqueName: \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.397130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.397114 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.398346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.398297 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.460120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.460086 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglp4\" (UniqueName: \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\") pod \"payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.565618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.565583 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:45.696914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.696875 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dff5711-32f8-421c-9115-d47205992f27" path="/var/lib/kubelet/pods/2dff5711-32f8-421c-9115-d47205992f27/volumes" Nov 02 23:35:45.949466 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:45.949397 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0e0549_0c5e_4a72_9814_46c74ac30bc2.slice/crio-21cf24c71fea51f656d9141e4469aac36e9651d2d3f5c3174c6496349830287b WatchSource:0}: Error finding container 21cf24c71fea51f656d9141e4469aac36e9651d2d3f5c3174c6496349830287b: Status 404 returned error can't find the container with id 21cf24c71fea51f656d9141e4469aac36e9651d2d3f5c3174c6496349830287b Nov 02 23:35:45.949727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:45.949708 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 02 23:35:46.118270 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.118232 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" event={"ID":"fc0e0549-0c5e-4a72-9814-46c74ac30bc2","Type":"ContainerStarted","Data":"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba"} Nov 02 23:35:46.118461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.118295 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" event={"ID":"fc0e0549-0c5e-4a72-9814-46c74ac30bc2","Type":"ContainerStarted","Data":"21cf24c71fea51f656d9141e4469aac36e9651d2d3f5c3174c6496349830287b"} Nov 02 23:35:46.118461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.118421 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:46.248759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.248653 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" podStartSLOduration=1.2486382520000001 podStartE2EDuration="1.248638252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:46.245998968 +0000 UTC m=+2860.675868828" watchObservedRunningTime="2026-11-02 23:35:46.248638252 +0000 UTC m=+2860.678508119" Nov 02 23:35:46.532504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.532420 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:35:46.537029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.537005 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.590472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.590441 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a5d6d8-serving-cert\"" Nov 02 23:35:46.596389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.596359 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:35:46.614016 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.613978 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.614210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.614025 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.614210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.614053 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjfv\" (UniqueName: \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.614210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.614164 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.703983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.703955 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:46.715410 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.715376 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.715556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.715437 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.715556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.715477 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjfv\" (UniqueName: \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.715556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.715517 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.715722 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:46.715594 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-serving-cert: secret "maas-api-e2e-praxis-b-a5d6d8-serving-cert" not found Nov 02 23:35:46.715722 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:46.715671 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls podName:6b93938d-0915-4e2e-a00a-a90b413a245d nodeName:}" failed. No retries permitted until 2026-11-02 23:35:47.215650058 +0000 UTC m=+2861.645519919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls") pod "maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" (UID: "6b93938d-0915-4e2e-a00a-a90b413a245d") : secret "maas-api-e2e-praxis-b-a5d6d8-serving-cert" not found Nov 02 23:35:46.716063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.716043 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.718235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.718217 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.777617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.777582 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjfv\" (UniqueName: \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:46.874843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:46.874757 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:47.028897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:47.028864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:47.223082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:47.223044 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:47.225921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:47.225889 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:47.452276 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:47.452239 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:47.575089 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:35:47.573713 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:35:47.841700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:47.841669 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:35:47.841908 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:35:47.841886 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b93938d_0915_4e2e_a00a_a90b413a245d.slice/crio-490e68cdc216257d37693a65c431b5b2cf31c5be742f178d1c004abd2dd1dc0b WatchSource:0}: Error finding container 490e68cdc216257d37693a65c431b5b2cf31c5be742f178d1c004abd2dd1dc0b: Status 404 returned error can't find the container with id 490e68cdc216257d37693a65c431b5b2cf31c5be742f178d1c004abd2dd1dc0b Nov 02 23:35:48.134450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:48.134363 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" event={"ID":"6b93938d-0915-4e2e-a00a-a90b413a245d","Type":"ContainerStarted","Data":"490e68cdc216257d37693a65c431b5b2cf31c5be742f178d1c004abd2dd1dc0b"} Nov 02 23:35:49.146815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:49.146718 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" event={"ID":"6b93938d-0915-4e2e-a00a-a90b413a245d","Type":"ContainerStarted","Data":"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f"} Nov 02 23:35:49.147318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:49.146950 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:49.269279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:49.269222 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" podStartSLOduration=2.4016352579999998 podStartE2EDuration="3.26920684s" totalImagesPullingTime="867.571582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:49.265037808 +0000 UTC m=+2863.694907668" watchObservedRunningTime="2026-11-02 23:35:49.26920684 +0000 UTC m=+2863.699076707" Nov 02 23:35:52.102582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:52.102539 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:35:55.160807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:55.160782 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:35:57.129494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:57.129465 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 02 23:35:57.459101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:35:57.459066 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:00.048024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:00.047991 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc"] Nov 02 23:36:00.098084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:00.098054 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e2684-dmkkc"] Nov 02 23:36:01.634532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:01.634494 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:36:01.634960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:01.634697 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" podUID="f128405e-66be-47da-bab5-b86f29836c33" containerName="maas-api" containerID="cri-o://9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd" gracePeriod=30 Nov 02 23:36:01.701495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:01.701456 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca73293b-b51c-4d51-b60b-22a58482f046" path="/var/lib/kubelet/pods/ca73293b-b51c-4d51-b60b-22a58482f046/volumes" Nov 02 23:36:01.738737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:01.738695 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:02.019817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.019418 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:36:02.166833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.161066 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\" (UniqueName: \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\") pod \"f128405e-66be-47da-bab5-b86f29836c33\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " Nov 02 23:36:02.166833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.161204 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\") pod \"f128405e-66be-47da-bab5-b86f29836c33\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " Nov 02 23:36:02.166833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.161463 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") pod \"f128405e-66be-47da-bab5-b86f29836c33\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " Nov 02 23:36:02.166833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.161524 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\") pod \"f128405e-66be-47da-bab5-b86f29836c33\" (UID: \"f128405e-66be-47da-bab5-b86f29836c33\") " Nov 02 23:36:02.172878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.167541 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca" pod "f128405e-66be-47da-bab5-b86f29836c33" (UID: "f128405e-66be-47da-bab5-b86f29836c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:02.199215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.199152 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs" pod "f128405e-66be-47da-bab5-b86f29836c33" (UID: "f128405e-66be-47da-bab5-b86f29836c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:02.199379 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.199273 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t" pod "f128405e-66be-47da-bab5-b86f29836c33" (UID: "f128405e-66be-47da-bab5-b86f29836c33"). InnerVolumeSpecName "kube-api-access-5mw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:02.207227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.205537 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls" pod "f128405e-66be-47da-bab5-b86f29836c33" (UID: "f128405e-66be-47da-bab5-b86f29836c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:02.264323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.264284 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mw9t\" (UniqueName: \"kubernetes.io/projected/f128405e-66be-47da-bab5-b86f29836c33-kube-api-access-5mw9t\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.264584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.264558 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.264725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.264587 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f128405e-66be-47da-bab5-b86f29836c33-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.264725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.264603 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f128405e-66be-47da-bab5-b86f29836c33-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:02.265898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.265864 4494 generic.go:356] "Generic (PLEG): container finished" podID="f128405e-66be-47da-bab5-b86f29836c33" containerID="9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd" exitCode=0 Nov 02 23:36:02.266045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.265985 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" Nov 02 23:36:02.266109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.266039 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" event={"ID":"f128405e-66be-47da-bab5-b86f29836c33","Type":"ContainerDied","Data":"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd"} Nov 02 23:36:02.266187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.266122 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726" event={"ID":"f128405e-66be-47da-bab5-b86f29836c33","Type":"ContainerDied","Data":"dae9e84c615b302696e197750e0ac3c3a2d8b17a9cd15274ffd72f0ef6171147"} Nov 02 23:36:02.266187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.266151 4494 scope.go:122] "RemoveContainer" containerID="9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd" Nov 02 23:36:02.282244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.281948 4494 scope.go:122] "RemoveContainer" containerID="9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd" Nov 02 23:36:02.283757 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:02.283694 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd\": container with ID starting with 9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd not found: ID does not exist" containerID="9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd" Nov 02 23:36:02.283879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.283772 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd"} err="failed to get container status \"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd\": rpc error: code = NotFound desc = could not find container \"9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd\": container with ID starting with 9c6cf08353d1d69c44d3d9bdf35469fca6bbc05658501f13ac4dbd931e9608fd not found: ID does not exist" Nov 02 23:36:02.284143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.284116 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:02.434767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.434729 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:02.438673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.438645 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:36:02.442122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:02.442101 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2279ad-65578c484b-gn726"] Nov 02 23:36:03.283981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.283955 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 02 23:36:03.284672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.284657 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f128405e-66be-47da-bab5-b86f29836c33" containerName="maas-api" Nov 02 23:36:03.284737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.284674 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f128405e-66be-47da-bab5-b86f29836c33" containerName="maas-api" Nov 02 23:36:03.284778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.284763 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f128405e-66be-47da-bab5-b86f29836c33" containerName="maas-api" Nov 02 23:36:03.288506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.288489 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.379315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.379275 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.379522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.379403 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.379611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.379529 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prcn\" (UniqueName: \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.379672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.379607 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.392378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.392351 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1f0009-serving-cert\"" Nov 02 23:36:03.396727 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.396703 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 02 23:36:03.480848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.480811 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.480848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.480846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:03.481095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.480884 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.481095 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:03.480954 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-serving-cert: secret "maas-api-e2e-shared-b-w3-1f0009-serving-cert" not found Nov 02 23:36:03.481095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.481003 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prcn\" (UniqueName: \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.481095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.481052 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.481095 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:03.481080 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls podName:5641f8cb-eb90-4227-b066-f9b7bc1f5581 nodeName:}" failed. No retries permitted until 2026-11-02 23:36:03.981050622 +0000 UTC m=+2878.410920472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls") pod "maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" (UID: "5641f8cb-eb90-4227-b066-f9b7bc1f5581") : secret "maas-api-e2e-shared-b-w3-1f0009-serving-cert" not found Nov 02 23:36:03.481589 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.481570 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.483957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.483932 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.571808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.571575 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prcn\" (UniqueName: \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:03.643466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.643430 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:03.754015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.747928 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f128405e-66be-47da-bab5-b86f29836c33" path="/var/lib/kubelet/pods/f128405e-66be-47da-bab5-b86f29836c33/volumes" Nov 02 23:36:03.787653 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:03.787615 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:04.015622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:04.015577 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:04.018280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:04.018257 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:04.203832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:04.203799 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:04.585446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:04.585418 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 02 23:36:04.585817 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:36:04.585736 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5641f8cb_eb90_4227_b066_f9b7bc1f5581.slice/crio-79058b8836e02898d4b6235e28507903806971d5c10e8daee803f1efc3460497 WatchSource:0}: Error finding container 79058b8836e02898d4b6235e28507903806971d5c10e8daee803f1efc3460497: Status 404 returned error can't find the container with id 79058b8836e02898d4b6235e28507903806971d5c10e8daee803f1efc3460497 Nov 02 23:36:04.887814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:04.887732 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:05.038095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:05.038061 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:05.049051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:05.049023 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck"] Nov 02 23:36:05.052508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:05.052486 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35cf3d2c-g66ck"] Nov 02 23:36:05.291758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:05.291721 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" event={"ID":"5641f8cb-eb90-4227-b066-f9b7bc1f5581","Type":"ContainerStarted","Data":"79058b8836e02898d4b6235e28507903806971d5c10e8daee803f1efc3460497"} Nov 02 23:36:05.696121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:05.696089 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc33012-d9d3-49ef-bb76-7ee7904fcd90" path="/var/lib/kubelet/pods/bfc33012-d9d3-49ef-bb76-7ee7904fcd90/volumes" Nov 02 23:36:06.300289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.300258 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" event={"ID":"5641f8cb-eb90-4227-b066-f9b7bc1f5581","Type":"ContainerStarted","Data":"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891"} Nov 02 23:36:06.300504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.300396 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:06.448784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.436069 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" podStartSLOduration=2.664612219 podStartE2EDuration="3.436040505s" totalImagesPullingTime="771.428286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:06.420341708 +0000 UTC m=+2880.850211569" watchObservedRunningTime="2026-11-02 23:36:06.436040505 +0000 UTC m=+2880.865910370" Nov 02 23:36:06.510466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.510427 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:06.944619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.944584 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:36:06.945148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.944862 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" containerID="cri-o://96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2" gracePeriod=30 Nov 02 23:36:06.981949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:06.981905 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:08.501012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.500974 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:08.757594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.757482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:08.865852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.865824 4494 scope.go:122] "RemoveContainer" containerID="e40ba40f90b494780d83078f3c6d57312378c0bb7b8485cc993ca64de3376b23" Nov 02 23:36:08.876464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.876441 4494 scope.go:122] "RemoveContainer" containerID="a0eb5707de0967f256692937f59679c705a64057a38f9dd7563ade0420f1ddf3" Nov 02 23:36:08.886572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.886553 4494 scope.go:122] "RemoveContainer" containerID="8b7b200c7414198ed80d8ef63dbc3eee32ec5888c897badce0998b289e2ef460" Nov 02 23:36:08.922192 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:08.922153 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:09.085250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.085018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:09.265650 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.265617 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:36:09.271889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.271868 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.345923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.345839 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4cdec0-openshift-default-dockercfg-4bjv9\"" Nov 02 23:36:09.352964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.352918 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:09.387022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.386986 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421097 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421294 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421346 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421392 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421523 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421580 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421630 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m59w\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421718 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.433196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.421746 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.522710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.522675 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.522753 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.522784 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.522880 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m59w\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.522977 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523007 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523104 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523144 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523159 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523366 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523424 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.523773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523634 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.524020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.523997 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.524118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.524102 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.526027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.526004 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.526349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.526330 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.603244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.602311 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:09.607165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.605033 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m59w\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.616220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.614590 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\") pod \"e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:09.886294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:09.886196 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:10.050973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.050940 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl"] Nov 02 23:36:10.054778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.054756 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5239cddb-zrgwl"] Nov 02 23:36:10.257956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.257922 4494 patch_prober.go:29] interesting pod/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:36:10.258140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.257989 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:36:10.329342 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:36:10.329302 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c17f1d_cfc3_48b5_9c9e_9acdcfab03f0.slice/crio-b130c4cd860c699913bd8ccfd6d4bdc46895dd93e2117a579a6215e489142810 WatchSource:0}: Error finding container b130c4cd860c699913bd8ccfd6d4bdc46895dd93e2117a579a6215e489142810: Status 404 returned error can't find the container with id b130c4cd860c699913bd8ccfd6d4bdc46895dd93e2117a579a6215e489142810 Nov 02 23:36:10.329538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.329461 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:36:10.332079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.332047 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:10.332158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.332113 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:10.332158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.332148 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:10.352794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:10.347556 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerStarted","Data":"b130c4cd860c699913bd8ccfd6d4bdc46895dd93e2117a579a6215e489142810"} Nov 02 23:36:11.356088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.356051 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerStarted","Data":"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9"} Nov 02 23:36:11.579383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.579336 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podStartSLOduration=2.579318203 podStartE2EDuration="2.579318203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:11.528642496 +0000 UTC m=+2885.958512360" watchObservedRunningTime="2026-11-02 23:36:11.579318203 +0000 UTC m=+2886.009188064" Nov 02 23:36:11.707555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.707520 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff6cccd-6cab-40ff-a517-894064a57bf3" path="/var/lib/kubelet/pods/4ff6cccd-6cab-40ff-a517-894064a57bf3/volumes" Nov 02 23:36:11.799715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.798468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:11.887899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.886831 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:11.894932 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:11.894901 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:12.234438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.234383 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:36:12.349671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.349644 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 02 23:36:12.365441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365411 4494 generic.go:356] "Generic (PLEG): container finished" podID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerID="96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2" exitCode=0 Nov 02 23:36:12.365930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365460 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" Nov 02 23:36:12.365930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365497 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerDied","Data":"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2"} Nov 02 23:36:12.365930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365549 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7" event={"ID":"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1","Type":"ContainerDied","Data":"3b2c5bc0feb0461369155fb2ee841ae7850e89473141eb5922a503c6292b7e70"} Nov 02 23:36:12.365930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365578 4494 scope.go:122] "RemoveContainer" containerID="96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2" Nov 02 23:36:12.365930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.365907 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:12.367044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.367025 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:36:12.379445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379419 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379461 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379558 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379620 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379602 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379671 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379769 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379796 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379847 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379861 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.379896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379874 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:12.380278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.379954 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\") pod \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\" (UID: \"95a5ffc8-7670-4cbe-b4b2-31b312b08fc1\") " Nov 02 23:36:12.380278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380070 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:12.380392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380275 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:12.380502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380476 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.380502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380507 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.380692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380521 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.380692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380535 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.380692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380550 4494 scope.go:122] "RemoveContainer" containerID="c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4" Nov 02 23:36:12.380868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.380761 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:12.383266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.383212 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:36:12.384044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.384015 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "kube-api-access-r6cxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:12.384044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.384030 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:12.390010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.389987 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token" pod "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" (UID: "95a5ffc8-7670-4cbe-b4b2-31b312b08fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:12.422621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.422601 4494 scope.go:122] "RemoveContainer" containerID="96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2" Nov 02 23:36:12.422963 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:12.422940 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2\": container with ID starting with 96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2 not found: ID does not exist" containerID="96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2" Nov 02 23:36:12.423031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.422972 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2"} err="failed to get container status \"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2\": rpc error: code = NotFound desc = could not find container \"96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2\": container with ID starting with 96f25ad37c6e8944b4f426e33aa0bc8e5c6b9f54ef09ebe5f6511d693434d2b2 not found: ID does not exist" Nov 02 23:36:12.423031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.422989 4494 scope.go:122] "RemoveContainer" containerID="c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4" Nov 02 23:36:12.423310 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:12.423292 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4\": container with ID starting with c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4 not found: ID does not exist" containerID="c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4" Nov 02 23:36:12.423365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.423317 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4"} err="failed to get container status \"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4\": rpc error: code = NotFound desc = could not find container \"c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4\": container with ID starting with c8c5f71152d51d4ad8924bbf92768a755d2005dbe8673d2ec03115edf136fcd4 not found: ID does not exist" Nov 02 23:36:12.482152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.482126 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.482152 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.482152 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.482372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.482166 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.482372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.482207 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cxn\" (UniqueName: \"kubernetes.io/projected/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-kube-api-access-r6cxn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.482372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.482217 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:36:12.854203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.852227 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:36:12.863615 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:12.863574 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2279ad-openshift-default-6f7f8466c9-g9rd7"] Nov 02 23:36:13.694244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:13.694215 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" path="/var/lib/kubelet/pods/95a5ffc8-7670-4cbe-b4b2-31b312b08fc1/volumes" Nov 02 23:36:14.277654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:14.277621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:14.636422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:14.636340 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:14.863525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:14.863492 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:15.010572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:15.010537 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:17.560677 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:17.560638 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:36:18.784220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:18.784184 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:18.971228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:18.971141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:19.047082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:19.046997 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b"] Nov 02 23:36:19.155469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:19.155431 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb2403f-hh42b"] Nov 02 23:36:19.668954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:19.668921 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:19.694384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:19.694352 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c431f8c-b020-4579-a3f3-99827e5c79b4" path="/var/lib/kubelet/pods/9c431f8c-b020-4579-a3f3-99827e5c79b4/volumes" Nov 02 23:36:25.047680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:25.047639 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:30.206348 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:30.206307 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:36:30.358306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:30.358277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:33.058108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:33.058061 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt"] Nov 02 23:36:33.100839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:33.100809 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c6df38-jlntt"] Nov 02 23:36:33.694881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:33.694832 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dc7054-aa78-401f-ab17-bdd975a07cb7" path="/var/lib/kubelet/pods/c3dc7054-aa78-401f-ab17-bdd975a07cb7/volumes" Nov 02 23:36:35.653200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:35.652722 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:40.404353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:40.404315 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:36:40.404751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:40.404388 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:36:40.635655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:40.635620 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:41.630666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.630577 4494 generic.go:356] "Generic (PLEG): container finished" podID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerID="59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965" exitCode=0 Nov 02 23:36:41.631134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.630654 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerDied","Data":"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965"} Nov 02 23:36:41.631749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.631726 4494 scope.go:122] "RemoveContainer" containerID="59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965" Nov 02 23:36:41.634803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.634753 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:41.634942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.634921 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:41.635050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:41.635019 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:36:42.658567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:42.658528 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerStarted","Data":"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4"} Nov 02 23:36:42.985416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:42.985380 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:36:42.985636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:42.985562 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:42.985636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:42.985606 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:43.985533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:43.985502 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:43.986022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:43.985561 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:44.985582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:44.985547 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:44.986060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:44.985602 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:45.059993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.059952 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr"] Nov 02 23:36:45.098255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.098220 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f9e0ba-mr4tr"] Nov 02 23:36:45.600040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.600010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:45.730256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.730218 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60301f04-3eb6-4ea9-bf60-152b7a9426ee" path="/var/lib/kubelet/pods/60301f04-3eb6-4ea9-bf60-152b7a9426ee/volumes" Nov 02 23:36:45.985363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.985321 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:45.985582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:45.985384 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:46.985120 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:46.985083 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:46.985569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:46.985136 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:47.594546 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:47.594461 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:36:47.985357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:47.985321 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body= Nov 02 23:36:47.985777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:47.985382 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" Nov 02 23:36:48.059758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:48.059719 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m"] Nov 02 23:36:48.100393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:48.100354 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8361cd2-r5s6m"] Nov 02 23:36:48.989522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:48.989494 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:36:48.989986 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:48.989973 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:36:48.990978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:48.990957 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 02 23:36:49.694675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:49.694644 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a1b41c-d517-4096-8f8d-8625adb02adb" path="/var/lib/kubelet/pods/f0a1b41c-d517-4096-8f8d-8625adb02adb/volumes" Nov 02 23:36:50.236185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.236145 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 02 23:36:50.237020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.236995 4494 cpu_manager.go:414] "containerMap: removing container" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.237140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.237022 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.237140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.237034 4494 cpu_manager.go:414] "containerMap: removing container" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.237140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.237041 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.237363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.237316 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.237363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.237334 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a5ffc8-7670-4cbe-b4b2-31b312b08fc1" containerName="istio-proxy" Nov 02 23:36:50.240366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.240346 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.344908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.344880 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b8d3cb-serving-cert\"" Nov 02 23:36:50.350294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.350265 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 02 23:36:50.389508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.389469 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.389715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.389619 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.389715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.389711 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4ql\" (UniqueName: \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.389834 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.389792 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.425382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.425339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:50.496250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.492128 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.496250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.492261 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.496250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.492375 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.496250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.492482 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4ql\" (UniqueName: \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.514196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.498933 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.514536 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:50.500604 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-serving-cert: secret "maas-api-e2e-shared-b-w2-b8d3cb-serving-cert" not found Nov 02 23:36:50.514744 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:36:50.514729 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls podName:967731db-2140-4a0c-a9d3-2272b862aa67 nodeName:}" failed. No retries permitted until 2026-11-02 23:36:51.014698992 +0000 UTC m=+2925.444568855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" (UID: "967731db-2140-4a0c-a9d3-2272b862aa67") : secret "maas-api-e2e-shared-b-w2-b8d3cb-serving-cert" not found Nov 02 23:36:50.523918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.523851 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.554781 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.554749 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4ql\" (UniqueName: \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:50.583008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.582972 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:50.744278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:50.740829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:51.104224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:51.104149 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:51.106984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:51.106953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:51.153375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:51.153342 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:51.534762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:51.534729 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 02 23:36:51.535515 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:36:51.535482 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967731db_2140_4a0c_a9d3_2272b862aa67.slice/crio-6751e05ace7baf1b0c0dbdeebbb9cd9210c6925e9928b8ec374581f4149c40ba WatchSource:0}: Error finding container 6751e05ace7baf1b0c0dbdeebbb9cd9210c6925e9928b8ec374581f4149c40ba: Status 404 returned error can't find the container with id 6751e05ace7baf1b0c0dbdeebbb9cd9210c6925e9928b8ec374581f4149c40ba Nov 02 23:36:51.804455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:51.804373 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" event={"ID":"967731db-2140-4a0c-a9d3-2272b862aa67","Type":"ContainerStarted","Data":"6751e05ace7baf1b0c0dbdeebbb9cd9210c6925e9928b8ec374581f4149c40ba"} Nov 02 23:36:52.814355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:52.814322 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" event={"ID":"967731db-2140-4a0c-a9d3-2272b862aa67","Type":"ContainerStarted","Data":"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381"} Nov 02 23:36:52.814792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:52.814395 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:36:52.935578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:52.935525 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" podStartSLOduration=2.090425711 podStartE2EDuration="2.935511392s" totalImagesPullingTime="845.085681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:52.93202617 +0000 UTC m=+2927.361896034" watchObservedRunningTime="2026-11-02 23:36:52.935511392 +0000 UTC m=+2927.365381253" Nov 02 23:36:55.665142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:55.665103 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:36:58.827505 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:36:58.827479 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 02 23:37:00.657884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:00.657805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:02.049498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:02.049464 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc"] Nov 02 23:37:02.097788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:02.097757 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32992acd-h25dc"] Nov 02 23:37:03.695068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:03.695034 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55d3d24-cce7-4797-8281-f3d9f780c21f" path="/var/lib/kubelet/pods/c55d3d24-cce7-4797-8281-f3d9f780c21f/volumes" Nov 02 23:37:05.049457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:05.049424 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw"] Nov 02 23:37:05.139213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:05.138841 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f6a640-dg2zw"] Nov 02 23:37:05.456637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:05.456590 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:37:05.695510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:05.695480 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bc166f-6881-4838-a3cd-0b470cd0d2dc" path="/var/lib/kubelet/pods/26bc166f-6881-4838-a3cd-0b470cd0d2dc/volumes" Nov 02 23:37:05.824472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:05.824391 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:08.224644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:08.224608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:08.957434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:08.957398 4494 scope.go:122] "RemoveContainer" containerID="9066994d6b458ffa6e144fa55ad42203fa712ffacee234a5b6770dea09ff3040" Nov 02 23:37:08.979090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:08.979063 4494 scope.go:122] "RemoveContainer" containerID="0b15f9de71896d511b712aefaacee631cc12ae1c0e4e9d4ccd5a2d23bb8aebc1" Nov 02 23:37:08.996286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:08.996257 4494 scope.go:122] "RemoveContainer" containerID="a2a65a80976fcc8608e89dcd33dfa9a4bc9e6f0c0f36e4b16955f85864127f21" Nov 02 23:37:09.006708 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:09.006690 4494 scope.go:122] "RemoveContainer" containerID="2359b82c2e555a5e67fb18cc7a9181784293eb9e893bdd5d7727b635f229c9eb" Nov 02 23:37:11.573250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:11.567233 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:12.145332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:12.145294 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:13.479126 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:13.479088 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:14.048407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:14.048369 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf"] Nov 02 23:37:14.101132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:14.101091 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d0ad79-tz9sf"] Nov 02 23:37:14.250003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:14.249530 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:37:14.250335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:14.250302 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:37:15.055094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.055063 4494 generic.go:356] "Generic (PLEG): container finished" podID="510b384f-58fa-415b-be77-feb8e7eb519c" containerID="0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf" exitCode=0 Nov 02 23:37:15.055546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.055137 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerDied","Data":"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf"} Nov 02 23:37:15.055638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.055623 4494 scope.go:122] "RemoveContainer" containerID="0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf" Nov 02 23:37:15.057380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.057352 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:15.057479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.057419 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:15.057479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.057462 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:15.696938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:15.695635 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32ab03e-6e23-422d-a581-451ac725dcce" path="/var/lib/kubelet/pods/b32ab03e-6e23-422d-a581-451ac725dcce/volumes" Nov 02 23:37:16.064516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.064420 4494 generic.go:356] "Generic (PLEG): container finished" podID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerID="181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945" exitCode=0 Nov 02 23:37:16.064516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.064494 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerDied","Data":"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945"} Nov 02 23:37:16.065006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.064988 4494 scope.go:122] "RemoveContainer" containerID="181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945" Nov 02 23:37:16.066854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.066827 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:16.066985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.066902 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:16.066985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.066833 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerStarted","Data":"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a"} Nov 02 23:37:16.066985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.066935 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:37:16.570236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.570196 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:16.987800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.987328 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:37:16.988147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.988126 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Nov 02 23:37:16.988475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:16.988336 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Nov 02 23:37:17.105214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.103164 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerStarted","Data":"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4"} Nov 02 23:37:17.319316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.319220 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:37:17.319484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.319420 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:17.319484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.319460 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:17.623785 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:37:17.623700 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:37:17.987036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.986997 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Nov 02 23:37:17.987244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:17.987055 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Nov 02 23:37:18.320023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:18.319936 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:18.320023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:18.319996 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:18.986428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:18.986394 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Nov 02 23:37:18.986629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:18.986452 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Nov 02 23:37:19.319283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:19.319194 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:19.319283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:19.319253 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:19.986638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:19.986606 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Nov 02 23:37:19.987073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:19.986659 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Nov 02 23:37:20.319883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:20.319802 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:20.319883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:20.319858 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:20.986738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:20.986692 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Nov 02 23:37:20.987420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:20.987379 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Nov 02 23:37:21.319848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.319758 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:21.319848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.319823 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:21.925437 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.925397 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:21.990459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.990425 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:37:21.990897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.990880 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:37:21.991637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:21.991617 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:37:22.319238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:22.319133 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 02 23:37:22.319520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:22.319482 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 02 23:37:23.046925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.046889 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2"] Nov 02 23:37:23.097393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.097364 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9672258a-95kn2"] Nov 02 23:37:23.323587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.323514 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:37:23.323999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.323971 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:37:23.324624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.324607 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 02 23:37:23.695700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:23.695665 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98e983f-877e-4a73-bc2d-8ccf8b615a31" path="/var/lib/kubelet/pods/a98e983f-877e-4a73-bc2d-8ccf8b615a31/volumes" Nov 02 23:37:24.545245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:24.545161 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:27.049027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.048987 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts"] Nov 02 23:37:27.099606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.099570 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a12212-76xts"] Nov 02 23:37:27.191757 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.191725 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:37:27.195448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.195427 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.249878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.249852 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4cdec0-serving-cert\"" Nov 02 23:37:27.254864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.254841 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:37:27.317241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.317147 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.317399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.317315 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqbf\" (UniqueName: \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.317495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.317479 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.317535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.317515 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.418732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.418690 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.418936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.418743 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.418936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.418812 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.418936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.418874 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqbf\" (UniqueName: \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.419135 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:37:27.418943 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-serving-cert: secret "maas-api-e2e-praxis-a-4cdec0-serving-cert" not found Nov 02 23:37:27.419135 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:37:27.419028 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls podName:c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7 nodeName:}" failed. No retries permitted until 2026-11-02 23:37:27.919007904 +0000 UTC m=+2962.348877763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls") pod "maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" (UID: "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7") : secret "maas-api-e2e-praxis-a-4cdec0-serving-cert" not found Nov 02 23:37:27.419507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.419483 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.421482 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.421460 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.426500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.426478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:27.480864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.480840 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqbf\" (UniqueName: \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.584506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.584414 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:27.702226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.702157 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6fd454a-d4f5-43d4-84bf-c74e709fbfc2" path="/var/lib/kubelet/pods/b6fd454a-d4f5-43d4-84bf-c74e709fbfc2/volumes" Nov 02 23:37:27.728978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.728941 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:27.941434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.941405 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:27.944006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:27.943971 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:28.109687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:28.109650 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:28.489265 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:37:28.489229 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d7b3fa_74e4_4f0e_ab91_0ce84651d2a7.slice/crio-5a664c0235bfe62ccf9d92f9b0ae0eb2629b5503095e2dc732008314f4b18c00 WatchSource:0}: Error finding container 5a664c0235bfe62ccf9d92f9b0ae0eb2629b5503095e2dc732008314f4b18c00: Status 404 returned error can't find the container with id 5a664c0235bfe62ccf9d92f9b0ae0eb2629b5503095e2dc732008314f4b18c00 Nov 02 23:37:28.490335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:28.490303 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:37:28.490867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:28.490852 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:37:29.224547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:29.224511 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" event={"ID":"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7","Type":"ContainerStarted","Data":"5a664c0235bfe62ccf9d92f9b0ae0eb2629b5503095e2dc732008314f4b18c00"} Nov 02 23:37:30.076406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:30.076371 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:30.233384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:30.233351 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" event={"ID":"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7","Type":"ContainerStarted","Data":"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55"} Nov 02 23:37:30.233875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:30.233469 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:30.367253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:30.357585 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" podStartSLOduration=2.5644895610000003 podStartE2EDuration="3.357548219s" totalImagesPullingTime="793.058658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:30.356561729 +0000 UTC m=+2964.786431584" watchObservedRunningTime="2026-11-02 23:37:30.357548219 +0000 UTC m=+2964.787418082" Nov 02 23:37:32.871767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:32.871730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:33.421969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:33.421935 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:37:36.290246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:36.290216 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:37:37.850398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:37.850358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:39.049666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:39.049632 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7"] Nov 02 23:37:39.100139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:39.100101 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83de46-qzhw7"] Nov 02 23:37:39.704331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:39.704296 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9deb39-7518-4ccb-aa22-7ae9b721a2bb" path="/var/lib/kubelet/pods/be9deb39-7518-4ccb-aa22-7ae9b721a2bb/volumes" Nov 02 23:37:42.734552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:42.734514 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:47.565724 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:37:47.565672 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:37:47.584717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:47.584672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:52.469042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:52.469007 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:37:57.488381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:37:57.488344 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:02.355220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.355186 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:02.531705 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.531672 4494 generic.go:356] "Generic (PLEG): container finished" podID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerID="d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9" exitCode=0 Nov 02 23:38:02.531891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.531747 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerDied","Data":"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9"} Nov 02 23:38:02.532220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.532202 4494 scope.go:122] "RemoveContainer" containerID="d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9" Nov 02 23:38:02.533782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.533756 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:02.533859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.533824 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:02.533906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:02.533857 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:03.564898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:03.564867 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerStarted","Data":"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895"} Nov 02 23:38:03.887326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:03.887234 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:38:03.887521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:03.887454 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 23:38:03.887601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:03.887520 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 23:38:04.099212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:04.099159 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:04.256681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:04.256641 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:04.886842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:04.886811 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 23:38:04.887283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:04.886863 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 23:38:05.887444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:05.887401 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 23:38:05.887846 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:05.887460 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 23:38:06.886920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:06.886883 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 23:38:06.887147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:06.886960 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 23:38:07.308504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:07.308460 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:07.886930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:07.886896 4494 patch_prober.go:29] interesting pod/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 02 23:38:07.887107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:07.886947 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 02 23:38:08.891137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:08.891106 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:38:08.891707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:08.891553 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:38:08.892453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:08.892435 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:38:09.108228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.108191 4494 scope.go:122] "RemoveContainer" containerID="406514dad2e64e11fdab1a4b1c0f880657ad763a04db7992b6d0a3cab16a2f96" Nov 02 23:38:09.119747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.119725 4494 scope.go:122] "RemoveContainer" containerID="de6fe94e819929a5509adcca72839f666d0270f57acca86ef0728c17ed7d404a" Nov 02 23:38:09.132425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.132402 4494 scope.go:122] "RemoveContainer" containerID="3fa271a93fb496442ff86448c97ab8f7e43e1e85306858ef19cb44b7825277da" Nov 02 23:38:09.142123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.142102 4494 scope.go:122] "RemoveContainer" containerID="84613dc434492f97e4d676852de5cffc521ac776127f9a0d54b01e4786e3c125" Nov 02 23:38:09.151900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.151881 4494 scope.go:122] "RemoveContainer" containerID="1ca19e37547719919c985e090475938a8714c15565edc4e705fafefeb6ce069a" Nov 02 23:38:09.161488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:09.161464 4494 scope.go:122] "RemoveContainer" containerID="14637b7e281c1de01e2b87713af5072e743b6f350e6a2710c69479cc82236d17" Nov 02 23:38:12.244524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:12.244489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:12.494380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:12.494347 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:13.357984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:13.357948 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:14.095587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:14.095553 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:17.548913 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:17.548798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:17.561912 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:17.561877 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:38:19.115968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:19.115928 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:38:22.502848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:22.502797 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:27.339185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:27.339147 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:29.713689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:29.713657 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:38:29.714111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:29.713887 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" containerID="cri-o://c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a" gracePeriod=30 Nov 02 23:38:29.764063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:29.764026 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:38:29.764273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:29.764235 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" containerID="cri-o://ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc" gracePeriod=30 Nov 02 23:38:32.242250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:32.242220 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:32.452621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:32.452584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:35.034416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.034378 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:38:35.076077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.076054 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:38:35.126097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126073 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.126295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126123 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.126295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126152 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.126295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126223 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.126295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126280 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126314 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126357 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126401 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126443 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126469 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126469 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.126572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126526 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126575 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126621 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126719 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126762 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126774 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126789 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126802 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126839 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.126970 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127057 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127149 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\") pod \"bb12f013-bf28-40bf-ae94-fc775ca78045\" (UID: \"bb12f013-bf28-40bf-ae94-fc775ca78045\") " Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127206 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\") pod \"c6264907-348a-4d7b-b652-aba5ea29b90b\" (UID: \"c6264907-348a-4d7b-b652-aba5ea29b90b\") " Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127407 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127441 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127499 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.127843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127647 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127946 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127966 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6264907-348a-4d7b-b652-aba5ea29b90b-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127981 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.127996 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb12f013-bf28-40bf-ae94-fc775ca78045-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128010 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128023 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128037 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128050 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128063 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128075 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.128959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.128932 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "kube-api-access-jzbpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.129333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.129313 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:35.129889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.129866 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "kube-api-access-kjbz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.129983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.129906 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.130073 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.130043 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token" pod "bb12f013-bf28-40bf-ae94-fc775ca78045" (UID: "bb12f013-bf28-40bf-ae94-fc775ca78045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.130417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.130396 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:35.130615 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.130599 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:35.131255 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.131237 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy" pod "c6264907-348a-4d7b-b652-aba5ea29b90b" (UID: "c6264907-348a-4d7b-b652-aba5ea29b90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:35.229481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229448 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229478 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbz9\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-kube-api-access-kjbz9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229489 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229500 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229509 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229518 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbpl\" (UniqueName: \"kubernetes.io/projected/c6264907-348a-4d7b-b652-aba5ea29b90b-kube-api-access-jzbpl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229527 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb12f013-bf28-40bf-ae94-fc775ca78045-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.229686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.229535 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6264907-348a-4d7b-b652-aba5ea29b90b-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:35.807499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.807469 4494 generic.go:356] "Generic (PLEG): container finished" podID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerID="c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a" exitCode=0 Nov 02 23:38:35.807672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.807529 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" Nov 02 23:38:35.807672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.807566 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" event={"ID":"bb12f013-bf28-40bf-ae94-fc775ca78045","Type":"ContainerDied","Data":"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a"} Nov 02 23:38:35.807672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.807636 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh" event={"ID":"bb12f013-bf28-40bf-ae94-fc775ca78045","Type":"ContainerDied","Data":"d878f9094a2f0c44da185a920269f727a6c105b0499fb4d1a3e446656ac9394b"} Nov 02 23:38:35.807672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.807666 4494 scope.go:122] "RemoveContainer" containerID="c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a" Nov 02 23:38:35.809034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.809009 4494 generic.go:356] "Generic (PLEG): container finished" podID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerID="ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc" exitCode=0 Nov 02 23:38:35.809155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.809050 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" Nov 02 23:38:35.809155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.809089 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" event={"ID":"c6264907-348a-4d7b-b652-aba5ea29b90b","Type":"ContainerDied","Data":"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc"} Nov 02 23:38:35.809155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.809131 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4" event={"ID":"c6264907-348a-4d7b-b652-aba5ea29b90b","Type":"ContainerDied","Data":"f5543a75d4e17bdb13e47687c293a16407d8f9b079dbf44d84832115b2b2911e"} Nov 02 23:38:35.817833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.817806 4494 scope.go:122] "RemoveContainer" containerID="c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a" Nov 02 23:38:35.818135 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:35.818114 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a\": container with ID starting with c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a not found: ID does not exist" containerID="c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a" Nov 02 23:38:35.818210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.818141 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a"} err="failed to get container status \"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a\": rpc error: code = NotFound desc = could not find container \"c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a\": container with ID starting with c02ec84cb8799a64d975d8972ca2187f19795f9bc16d3f718fb684faa6278a8a not found: ID does not exist" Nov 02 23:38:35.818210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.818157 4494 scope.go:122] "RemoveContainer" containerID="ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc" Nov 02 23:38:35.827310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.827293 4494 scope.go:122] "RemoveContainer" containerID="ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc" Nov 02 23:38:35.827595 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:35.827578 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc\": container with ID starting with ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc not found: ID does not exist" containerID="ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc" Nov 02 23:38:35.827668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:35.827602 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc"} err="failed to get container status \"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc\": rpc error: code = NotFound desc = could not find container \"ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc\": container with ID starting with ee679a06eec57ef9bb9285ad66df94e6a6b0a8fd3756dc64712ba2d57ad274bc not found: ID does not exist" Nov 02 23:38:36.026316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:36.026194 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:38:36.031278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:36.031253 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-zjlr4"] Nov 02 23:38:36.141910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:36.141873 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:38:36.145430 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:36.145410 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-q5grh"] Nov 02 23:38:37.244766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:37.244732 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:37.695291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:37.695259 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" path="/var/lib/kubelet/pods/bb12f013-bf28-40bf-ae94-fc775ca78045/volumes" Nov 02 23:38:37.695785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:37.695770 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" path="/var/lib/kubelet/pods/c6264907-348a-4d7b-b652-aba5ea29b90b/volumes" Nov 02 23:38:42.090165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.090132 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:42.164799 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.164567 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:38:42.165415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.165381 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" containerID="cri-o://1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0" gracePeriod=30 Nov 02 23:38:42.210111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.210063 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:38:42.210376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.210326 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" containerID="cri-o://44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3" gracePeriod=30 Nov 02 23:38:42.213940 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.213904 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:38:42.214117 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:42.214098 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" containerID="cri-o://1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a" gracePeriod=30 Nov 02 23:38:43.180670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:43.180637 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:47.073158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.073121 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:47.507347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.507310 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:38:47.536202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.534147 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:38:47.598060 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:47.597963 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:38:47.609560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.609530 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:38:47.649851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.649825 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.650062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.649881 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.650062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.649946 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.650062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.649976 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.650062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.649998 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.650062 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650027 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650066 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650090 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650119 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650148 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650193 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650228 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650283 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650321 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650316 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650325 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.650396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650390 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650444 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650489 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650499 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650511 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650486 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650593 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650643 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650702 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650745 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.650822 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.651048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651028 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651075 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\") pod \"97ce74df-ae18-4e93-abfe-e8e200c41b08\" (UID: \"97ce74df-ae18-4e93-abfe-e8e200c41b08\") " Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651111 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651133 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651206 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\") pod \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\" (UID: \"0482ab7b-2cd7-4227-99cf-eb78059e5b07\") " Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651282 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\") pod \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\" (UID: \"0a0ef8d2-5de0-4016-86a6-fada53414ae8\") " Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651466 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.651648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651561 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.652026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651789 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.652026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.651796 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.652026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652012 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652033 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652047 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652059 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652071 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652085 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652098 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652112 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652126 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652142 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.652950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.652920 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.653115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653085 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.653279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653153 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:47.653493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653466 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:47.653573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653487 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:38:47.653731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.653993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.653968 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.654918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.654887 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.655495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.655471 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "kube-api-access-klxh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.655624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.655602 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:47.655739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.655717 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:38:47.655900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.655863 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.656003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.655985 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.656383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.656367 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token" pod "0482ab7b-2cd7-4227-99cf-eb78059e5b07" (UID: "0482ab7b-2cd7-4227-99cf-eb78059e5b07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.656606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.656587 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "kube-api-access-7j2t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.657154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.657131 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy" pod "97ce74df-ae18-4e93-abfe-e8e200c41b08" (UID: "97ce74df-ae18-4e93-abfe-e8e200c41b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:38:47.657269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.657251 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr" pod "0a0ef8d2-5de0-4016-86a6-fada53414ae8" (UID: "0a0ef8d2-5de0-4016-86a6-fada53414ae8"). InnerVolumeSpecName "kube-api-access-wmwxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:38:47.710312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.710280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:47.753084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753050 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753082 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klxh5\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-kube-api-access-klxh5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753093 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753103 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753112 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwxr\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-kube-api-access-wmwxr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753120 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753129 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753147 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753156 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j2t5\" (UniqueName: \"kubernetes.io/projected/97ce74df-ae18-4e93-abfe-e8e200c41b08-kube-api-access-7j2t5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753164 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ce74df-ae18-4e93-abfe-e8e200c41b08-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753187 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753195 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ce74df-ae18-4e93-abfe-e8e200c41b08-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753203 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ce74df-ae18-4e93-abfe-e8e200c41b08-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753213 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0482ab7b-2cd7-4227-99cf-eb78059e5b07-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753221 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0482ab7b-2cd7-4227-99cf-eb78059e5b07-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753231 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.753312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.753238 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0ef8d2-5de0-4016-86a6-fada53414ae8-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:38:47.898424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.898335 4494 generic.go:356] "Generic (PLEG): container finished" podID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerID="1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a" exitCode=0 Nov 02 23:38:47.898424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.898384 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" Nov 02 23:38:47.898690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.898426 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" event={"ID":"0a0ef8d2-5de0-4016-86a6-fada53414ae8","Type":"ContainerDied","Data":"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a"} Nov 02 23:38:47.898690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.898492 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk" event={"ID":"0a0ef8d2-5de0-4016-86a6-fada53414ae8","Type":"ContainerDied","Data":"05070ccea9eb094772763325d93f28e7f0ae6d68f10a1fbc792aa999c630ab49"} Nov 02 23:38:47.898690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.898523 4494 scope.go:122] "RemoveContainer" containerID="1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a" Nov 02 23:38:47.900032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.900011 4494 generic.go:356] "Generic (PLEG): container finished" podID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerID="1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0" exitCode=0 Nov 02 23:38:47.900154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.900079 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" Nov 02 23:38:47.900154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.900083 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" event={"ID":"0482ab7b-2cd7-4227-99cf-eb78059e5b07","Type":"ContainerDied","Data":"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0"} Nov 02 23:38:47.900295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.900151 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28" event={"ID":"0482ab7b-2cd7-4227-99cf-eb78059e5b07","Type":"ContainerDied","Data":"3393a87fe2a8dd9f5d468dbfae4525fca32ffe8fc739b03ac929b1b59f3fc66c"} Nov 02 23:38:47.901955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.901928 4494 generic.go:356] "Generic (PLEG): container finished" podID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerID="44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3" exitCode=0 Nov 02 23:38:47.902059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.901955 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" event={"ID":"97ce74df-ae18-4e93-abfe-e8e200c41b08","Type":"ContainerDied","Data":"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3"} Nov 02 23:38:47.902059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.902016 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" Nov 02 23:38:47.902059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.902023 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv" event={"ID":"97ce74df-ae18-4e93-abfe-e8e200c41b08","Type":"ContainerDied","Data":"dcff14122b3516cdac5e270acf7ec8ae12708e093451f269a7157101ff9abe12"} Nov 02 23:38:47.910763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.910740 4494 scope.go:122] "RemoveContainer" containerID="1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a" Nov 02 23:38:47.911086 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:47.911053 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a\": container with ID starting with 1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a not found: ID does not exist" containerID="1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a" Nov 02 23:38:47.911219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.911099 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a"} err="failed to get container status \"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a\": rpc error: code = NotFound desc = could not find container \"1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a\": container with ID starting with 1c06e79c6b7e08d5d1a10c3fecb934dff435cb49e0da5f7541dc105505ded23a not found: ID does not exist" Nov 02 23:38:47.911219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.911120 4494 scope.go:122] "RemoveContainer" containerID="1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0" Nov 02 23:38:47.929241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.929214 4494 scope.go:122] "RemoveContainer" containerID="1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0" Nov 02 23:38:47.929650 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:47.929616 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0\": container with ID starting with 1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0 not found: ID does not exist" containerID="1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0" Nov 02 23:38:47.929778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.929651 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0"} err="failed to get container status \"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0\": rpc error: code = NotFound desc = could not find container \"1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0\": container with ID starting with 1a510e2b5e87821269e857f47d01be34d0b90d8f36aaddd8c79196828a548da0 not found: ID does not exist" Nov 02 23:38:47.929778 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.929670 4494 scope.go:122] "RemoveContainer" containerID="44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3" Nov 02 23:38:47.940258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.940236 4494 scope.go:122] "RemoveContainer" containerID="44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3" Nov 02 23:38:47.940549 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:38:47.940528 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3\": container with ID starting with 44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3 not found: ID does not exist" containerID="44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3" Nov 02 23:38:47.940623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:47.940557 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3"} err="failed to get container status \"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3\": rpc error: code = NotFound desc = could not find container \"44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3\": container with ID starting with 44229d06b21b892e424f74fc6f73c8ec66e6ef85c9ac5c4767b1ea8a6d3610c3 not found: ID does not exist" Nov 02 23:38:48.062467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.062430 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:38:48.065632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.065609 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-84lqk"] Nov 02 23:38:48.177792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.177749 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:38:48.180975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.180953 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-2s9lv"] Nov 02 23:38:48.326407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.326373 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:38:48.376668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.376636 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-gbh28"] Nov 02 23:38:48.882395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:48.882362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:49.402004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.401972 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:38:49.402716 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402696 4494 cpu_manager.go:414] "containerMap: removing container" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402720 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402741 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402749 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402761 4494 cpu_manager.go:414] "containerMap: removing container" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402768 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" Nov 02 23:38:49.402811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402809 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" Nov 02 23:38:49.403041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402817 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" Nov 02 23:38:49.403041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402844 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" Nov 02 23:38:49.403041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.402850 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" Nov 02 23:38:49.403041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.403020 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" containerName="istio-proxy" Nov 02 23:38:49.403041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.403041 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" containerName="istio-proxy" Nov 02 23:38:49.403222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.403057 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb12f013-bf28-40bf-ae94-fc775ca78045" containerName="istio-proxy" Nov 02 23:38:49.403222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.403064 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6264907-348a-4d7b-b652-aba5ea29b90b" containerName="istio-proxy" Nov 02 23:38:49.403222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.403071 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" containerName="istio-proxy" Nov 02 23:38:49.408290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.408270 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.473047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.472986 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-dbe7b2-openshift-default-dockercfg-lmhqg\"" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.503038 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516574 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8qn\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516628 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516739 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516799 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.516900 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.517041 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.517104 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.579284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.517156 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661816 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661864 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661897 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8qn\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661937 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661969 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.661993 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.662027 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.662083 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.662160 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.662740 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.662905 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.663525 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.664061 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.664404 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.665199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.664641 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.666346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.666048 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.672205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.669699 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.694808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.694778 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0482ab7b-2cd7-4227-99cf-eb78059e5b07" path="/var/lib/kubelet/pods/0482ab7b-2cd7-4227-99cf-eb78059e5b07/volumes" Nov 02 23:38:49.695302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.695287 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0ef8d2-5de0-4016-86a6-fada53414ae8" path="/var/lib/kubelet/pods/0a0ef8d2-5de0-4016-86a6-fada53414ae8/volumes" Nov 02 23:38:49.695744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.695728 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ce74df-ae18-4e93-abfe-e8e200c41b08" path="/var/lib/kubelet/pods/97ce74df-ae18-4e93-abfe-e8e200c41b08/volumes" Nov 02 23:38:49.727827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.727795 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.727979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.727843 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8qn\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\") pod \"e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:49.787113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:49.787077 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:50.024761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.024682 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:50.462386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.462360 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:38:50.462870 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:38:50.462848 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b5f827_5dab_4336_96f3_9279ce9f60bd.slice/crio-45615cbfd840d97618e84bc581f2a8828120eaf5e3897e9ae37bd972507b8d54 WatchSource:0}: Error finding container 45615cbfd840d97618e84bc581f2a8828120eaf5e3897e9ae37bd972507b8d54: Status 404 returned error can't find the container with id 45615cbfd840d97618e84bc581f2a8828120eaf5e3897e9ae37bd972507b8d54 Nov 02 23:38:50.465403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.465371 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:50.465496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.465443 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:50.465496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.465477 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:38:50.926341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.926303 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerStarted","Data":"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386"} Nov 02 23:38:50.926529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:50.926359 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerStarted","Data":"45615cbfd840d97618e84bc581f2a8828120eaf5e3897e9ae37bd972507b8d54"} Nov 02 23:38:51.025598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:51.025569 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:51.031806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:51.031780 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:51.049409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:51.049354 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podStartSLOduration=2.049340738 podStartE2EDuration="2.049340738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:51.047822595 +0000 UTC m=+3045.477692470" watchObservedRunningTime="2026-11-02 23:38:51.049340738 +0000 UTC m=+3045.479210600" Nov 02 23:38:51.936045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:51.936014 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:51.936976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:51.936957 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:38:52.460398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:52.460360 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:54.031690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:54.031652 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:54.194122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:54.194087 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:54.341031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:54.340932 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:55.311229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:55.311191 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:38:57.266438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.266403 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:57.440922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.440891 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:57.676465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.676433 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:38:57.681093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.681070 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.737494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.737463 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:38:57.803245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.800550 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-dbe7b2\"" Nov 02 23:38:57.803245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.801935 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-dbe7b2-dockercfg-dmpsp\"" Nov 02 23:38:57.822196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.813751 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.822196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.813908 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnt7z\" (UniqueName: \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.873868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.873839 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:38:57.879443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.879415 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:57.899651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.899618 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:38:57.916525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.916496 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgq4b\" (UniqueName: \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:57.916737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.916708 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:57.916839 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.916821 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.916957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.916943 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt7z\" (UniqueName: \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.917748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.917720 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.979467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.979394 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnt7z\" (UniqueName: \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\") pod \"payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:57.997461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:57.997427 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:58.017737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.017709 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.017870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.017835 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgq4b\" (UniqueName: \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.018486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.018466 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.061191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.061121 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:38:58.081661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.081598 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgq4b\" (UniqueName: \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\") pod \"payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.205187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.205145 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.379006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.378980 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:38:58.380120 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:38:58.380098 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6117a085_64e6_4ecf_a781_ba89faba6cc0.slice/crio-aaf56ce339c1036dc730e724f8c240a70b7f86be5d77cb9ee9c450238d42ec12 WatchSource:0}: Error finding container aaf56ce339c1036dc730e724f8c240a70b7f86be5d77cb9ee9c450238d42ec12: Status 404 returned error can't find the container with id aaf56ce339c1036dc730e724f8c240a70b7f86be5d77cb9ee9c450238d42ec12 Nov 02 23:38:58.590296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.590266 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:38:58.591708 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:38:58.591683 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1636881_54c7_4da7_b4f3_c4330cffac58.slice/crio-c4421dfcd5cfb762f4e83ad7cebd128814f847cf674d2b1895f845c00331cef5 WatchSource:0}: Error finding container c4421dfcd5cfb762f4e83ad7cebd128814f847cf674d2b1895f845c00331cef5: Status 404 returned error can't find the container with id c4421dfcd5cfb762f4e83ad7cebd128814f847cf674d2b1895f845c00331cef5 Nov 02 23:38:58.987955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.987904 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" event={"ID":"6117a085-64e6-4ecf-a781-ba89faba6cc0","Type":"ContainerStarted","Data":"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04"} Nov 02 23:38:58.988213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.987992 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" event={"ID":"6117a085-64e6-4ecf-a781-ba89faba6cc0","Type":"ContainerStarted","Data":"aaf56ce339c1036dc730e724f8c240a70b7f86be5d77cb9ee9c450238d42ec12"} Nov 02 23:38:58.988213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.988184 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:38:58.989910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.989883 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" event={"ID":"f1636881-54c7-4da7-b4f3-c4330cffac58","Type":"ContainerStarted","Data":"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3"} Nov 02 23:38:58.990038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.989941 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:38:58.990038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:58.989963 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" event={"ID":"f1636881-54c7-4da7-b4f3-c4330cffac58","Type":"ContainerStarted","Data":"c4421dfcd5cfb762f4e83ad7cebd128814f847cf674d2b1895f845c00331cef5"} Nov 02 23:38:59.106027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:59.105958 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" podStartSLOduration=2.105939718 podStartE2EDuration="2.105939718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:59.104470804 +0000 UTC m=+3053.534340666" watchObservedRunningTime="2026-11-02 23:38:59.105939718 +0000 UTC m=+3053.535809582" Nov 02 23:38:59.255651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:38:59.252294 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" podStartSLOduration=2.252271983 podStartE2EDuration="2.252271983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:59.235656634 +0000 UTC m=+3053.665526495" watchObservedRunningTime="2026-11-02 23:38:59.252271983 +0000 UTC m=+3053.682141845" Nov 02 23:39:01.318508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:01.318468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:03.039032 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:03.038990 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:06.381323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:06.381291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:07.889350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:07.889318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:09.243095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:09.243054 4494 scope.go:122] "RemoveContainer" containerID="5b92d8a49437dc02358f402f4af56b4c5caee8b12acf486911afa78b9d754174" Nov 02 23:39:09.253449 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:09.253426 4494 scope.go:122] "RemoveContainer" containerID="34eb2990b005e21d5f4d46caa8f1ce06f6a59a9807aff136f04d68c6b317cd22" Nov 02 23:39:09.262453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:09.262432 4494 scope.go:122] "RemoveContainer" containerID="4a5c35afbc5ef7729bc3752e923a869934fd5c6e7c093d8536f68a6d1c9c1a47" Nov 02 23:39:09.271786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:09.271767 4494 scope.go:122] "RemoveContainer" containerID="5c4aa1541f08d54cb6a643fbc10d41adf5602e1d96732f65ec75e8c5d7727052" Nov 02 23:39:10.000009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:09.999982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:39:10.000199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:10.000044 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:39:12.741789 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:12.741758 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:12.884842 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:12.884802 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:17.567999 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:39:17.567958 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:39:17.597316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:17.597283 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:19.129426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:19.129394 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:19.272944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:19.272909 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:22.560905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:22.560829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:27.477465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:27.477431 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:32.445240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:32.445207 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:35.812497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:35.812458 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:37.344384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:37.344294 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:42.236617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:42.236583 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:42.588575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:42.588487 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:39:42.703377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:42.703343 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:39:42.733673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:42.733639 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:42.954235 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:42.954201 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:43.098750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:43.098682 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:44.220127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:44.220096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:47.540007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:47.539919 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:47.564678 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:39:47.564585 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:39:47.594970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:47.594927 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" containerID="cri-o://db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67" gracePeriod=25 Nov 02 23:39:47.704077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:47.704038 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" containerID="cri-o://7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466" gracePeriod=25 Nov 02 23:39:49.339579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:49.339530 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:50.088427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:50.088388 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:52.400613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:52.400577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:52.801484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:52.801446 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:39:53.813876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:53.813828 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:39:55.681664 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:55.681625 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:57.284920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:57.284881 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:39:59.491050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:39:59.491019 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:40:02.234197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:02.234130 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:02.801400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:02.801366 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:40:03.813869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:03.813829 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:40:06.896308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:06.896271 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:40:06.900898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:06.900879 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:06.955621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:06.955585 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-dbe7b2-serving-cert\"" Nov 02 23:40:06.959758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:06.959727 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:40:07.029562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.029523 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.029562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.029575 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9rd\" (UniqueName: \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.029821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.029713 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.029880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.029853 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.088189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.088141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:07.130695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.130657 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.130695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.130695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9rd\" (UniqueName: \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.130912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.130759 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.130912 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:07.130805 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-serving-cert: secret "maas-api-e2e-praxis-b-dbe7b2-serving-cert" not found Nov 02 23:40:07.130912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.130827 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.130912 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:07.130861 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls podName:4cefe6d3-ef25-44ca-89d6-cecc54a0519b nodeName:}" failed. No retries permitted until 2026-11-02 23:40:07.630841644 +0000 UTC m=+3122.060711486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls") pod "maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" (UID: "4cefe6d3-ef25-44ca-89d6-cecc54a0519b") : secret "maas-api-e2e-praxis-b-dbe7b2-serving-cert" not found Nov 02 23:40:07.131462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.131445 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.133429 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.133413 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.194920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.194897 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9rd\" (UniqueName: \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.248066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.248034 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:07.398609 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.398576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:07.552878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.552763 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:07.640031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.640003 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.642497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.642476 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.820526 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.820437 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:07.867226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:07.867161 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:40:08.045879 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.045851 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\") pod \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " Nov 02 23:40:08.046343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.046047 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\" (UniqueName: \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\") pod \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\" (UID: \"6d96b446-1ec0-45a7-9358-a5d9e31ee69e\") " Nov 02 23:40:08.046343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.046323 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume" pod "6d96b446-1ec0-45a7-9358-a5d9e31ee69e" (UID: "6d96b446-1ec0-45a7-9358-a5d9e31ee69e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:08.046719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.046685 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:08.048503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.048481 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf" pod "6d96b446-1ec0-45a7-9358-a5d9e31ee69e" (UID: "6d96b446-1ec0-45a7-9358-a5d9e31ee69e"). InnerVolumeSpecName "kube-api-access-g27zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:08.057104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.057080 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:40:08.147440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.147413 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\") pod \"668c3b82-4242-43eb-946f-78a802c34d4e\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " Nov 02 23:40:08.147640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.147626 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\" (UniqueName: \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\") pod \"668c3b82-4242-43eb-946f-78a802c34d4e\" (UID: \"668c3b82-4242-43eb-946f-78a802c34d4e\") " Nov 02 23:40:08.147808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.147780 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume" pod "668c3b82-4242-43eb-946f-78a802c34d4e" (UID: "668c3b82-4242-43eb-946f-78a802c34d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:08.148095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.148072 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27zf\" (UniqueName: \"kubernetes.io/projected/6d96b446-1ec0-45a7-9358-a5d9e31ee69e-kube-api-access-g27zf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:08.148095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.148099 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668c3b82-4242-43eb-946f-78a802c34d4e-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:08.150021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.149997 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk" pod "668c3b82-4242-43eb-946f-78a802c34d4e" (UID: "668c3b82-4242-43eb-946f-78a802c34d4e"). InnerVolumeSpecName "kube-api-access-svvmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:08.249700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.249670 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvmk\" (UniqueName: \"kubernetes.io/projected/668c3b82-4242-43eb-946f-78a802c34d4e-kube-api-access-svvmk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:08.295429 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:40:08.295394 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cefe6d3_ef25_44ca_89d6_cecc54a0519b.slice/crio-51596a506a86ebd6643c62ef5a2535492bb54f3ea798f8402e89a818ce073922 WatchSource:0}: Error finding container 51596a506a86ebd6643c62ef5a2535492bb54f3ea798f8402e89a818ce073922: Status 404 returned error can't find the container with id 51596a506a86ebd6643c62ef5a2535492bb54f3ea798f8402e89a818ce073922 Nov 02 23:40:08.297229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.297205 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:40:08.549954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.549912 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" event={"ID":"4cefe6d3-ef25-44ca-89d6-cecc54a0519b","Type":"ContainerStarted","Data":"51596a506a86ebd6643c62ef5a2535492bb54f3ea798f8402e89a818ce073922"} Nov 02 23:40:08.551362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.551328 4494 generic.go:356] "Generic (PLEG): container finished" podID="668c3b82-4242-43eb-946f-78a802c34d4e" containerID="7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466" exitCode=0 Nov 02 23:40:08.551512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.551384 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" Nov 02 23:40:08.551512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.551416 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" event={"ID":"668c3b82-4242-43eb-946f-78a802c34d4e","Type":"ContainerDied","Data":"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466"} Nov 02 23:40:08.551512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.551470 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn" event={"ID":"668c3b82-4242-43eb-946f-78a802c34d4e","Type":"ContainerDied","Data":"897dccb614aeee332e1421dfa5d25a3acd278e9f926da0f32dd1aba54efef720"} Nov 02 23:40:08.551512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.551500 4494 scope.go:122] "RemoveContainer" containerID="7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466" Nov 02 23:40:08.553254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.553229 4494 generic.go:356] "Generic (PLEG): container finished" podID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerID="db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67" exitCode=0 Nov 02 23:40:08.553387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.553269 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" Nov 02 23:40:08.553387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.553276 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" event={"ID":"6d96b446-1ec0-45a7-9358-a5d9e31ee69e","Type":"ContainerDied","Data":"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67"} Nov 02 23:40:08.553387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.553318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl" event={"ID":"6d96b446-1ec0-45a7-9358-a5d9e31ee69e","Type":"ContainerDied","Data":"f7b953752aa3a2591d679ca230426327da9fe669509919cf25a7a1df3f3f8438"} Nov 02 23:40:08.563409 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.563392 4494 scope.go:122] "RemoveContainer" containerID="7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466" Nov 02 23:40:08.563716 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:08.563692 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466\": container with ID starting with 7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466 not found: ID does not exist" containerID="7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466" Nov 02 23:40:08.563802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.563723 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466"} err="failed to get container status \"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466\": rpc error: code = NotFound desc = could not find container \"7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466\": container with ID starting with 7f94966f16a97d337ab649c7b7ac476ee5e0e5ea67ed916f801e7061aee1c466 not found: ID does not exist" Nov 02 23:40:08.563802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.563748 4494 scope.go:122] "RemoveContainer" containerID="db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67" Nov 02 23:40:08.576875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.576849 4494 scope.go:122] "RemoveContainer" containerID="db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67" Nov 02 23:40:08.577207 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:08.577158 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67\": container with ID starting with db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67 not found: ID does not exist" containerID="db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67" Nov 02 23:40:08.577308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.577206 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67"} err="failed to get container status \"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67\": rpc error: code = NotFound desc = could not find container \"db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67\": container with ID starting with db1b9db631e12539c52b28dc2fb9d889f06e00ee5fcac587e0b62b889b89ad67 not found: ID does not exist" Nov 02 23:40:08.719142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.719109 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:40:08.722645 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.722615 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-94e3ae-7c996dd765-cg8hl"] Nov 02 23:40:08.831623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.831454 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:40:08.835791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:08.835766 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-94e3ae-84bb94c55f-5qdnn"] Nov 02 23:40:09.561980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:09.561942 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" event={"ID":"4cefe6d3-ef25-44ca-89d6-cecc54a0519b","Type":"ContainerStarted","Data":"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984"} Nov 02 23:40:09.562450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:09.562007 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:09.681217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:09.681145 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" podStartSLOduration=2.655952224 podStartE2EDuration="3.681130285s" totalImagesPullingTime="1.025178061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:09.67988847 +0000 UTC m=+3124.109758337" watchObservedRunningTime="2026-11-02 23:40:09.681130285 +0000 UTC m=+3124.111000187" Nov 02 23:40:09.695209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:09.695166 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" path="/var/lib/kubelet/pods/668c3b82-4242-43eb-946f-78a802c34d4e/volumes" Nov 02 23:40:09.695622 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:09.695609 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" path="/var/lib/kubelet/pods/6d96b446-1ec0-45a7-9358-a5d9e31ee69e/volumes" Nov 02 23:40:12.236652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:12.236608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:13.193651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:13.193619 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:15.575225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:15.575191 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:40:17.115106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:17.115068 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:17.564115 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:17.564075 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:40:21.398734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:21.398705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:21.546425 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:21.546385 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:21.951697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:21.951658 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:26.955734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:26.955679 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:31.258578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:31.258526 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:32.978849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:32.978811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:33.200851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.200814 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:33.702923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.702892 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:40:33.703154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.703130 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" containerID="cri-o://4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f" gracePeriod=30 Nov 02 23:40:33.710353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.710327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:33.771854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.771814 4494 generic.go:356] "Generic (PLEG): container finished" podID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerID="0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386" exitCode=0 Nov 02 23:40:33.772080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.772046 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerDied","Data":"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386"} Nov 02 23:40:33.773762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.773734 4494 scope.go:122] "RemoveContainer" containerID="0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386" Nov 02 23:40:33.793203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.776042 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:40:33.793203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.776134 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:40:33.793203 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:33.776206 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:40:34.491232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:34.491198 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:34.781158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:34.781070 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerStarted","Data":"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5"} Nov 02 23:40:35.025847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:35.025814 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:40:35.027241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:35.027209 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:35.027432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:35.027264 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:36.025091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:36.025056 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:36.025511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:36.025122 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:36.279465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:36.279378 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:36.427694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:36.427657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:36.942291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:36.942256 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:40:37.025328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:37.025294 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:37.025725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:37.025345 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:38.025371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:38.025335 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:38.025779 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:38.025391 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:38.330047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:38.329964 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:39.025498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.025470 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:39.025823 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.025528 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:39.063783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.063749 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:40:39.117788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.117754 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.117788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.117794 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.117827 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.117867 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118000 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118046 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118139 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118251 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118308 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\") pod \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\" (UID: \"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff\") " Nov 02 23:40:39.118470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118246 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:39.118470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118458 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:39.118548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118271 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:39.118593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118569 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:39.118593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118580 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:39.118911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118888 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.118911 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118909 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.119119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118919 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.119119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118929 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.119119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.118937 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.120450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.120425 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:40:39.120596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.120572 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "kube-api-access-sddvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:39.120679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.120647 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:39.120841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.120823 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token" pod "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" (UID: "5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:39.220316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.220282 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.220316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.220309 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sddvv\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-kube-api-access-sddvv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.220316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.220318 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.220316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.220327 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:39.821666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.821635 4494 generic.go:356] "Generic (PLEG): container finished" podID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerID="4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f" exitCode=0 Nov 02 23:40:39.821850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.821689 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" Nov 02 23:40:39.821850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.821708 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerDied","Data":"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f"} Nov 02 23:40:39.821850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.821751 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9" event={"ID":"5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff","Type":"ContainerDied","Data":"e0c32c05ce0eaa4cbc728f1e1e88e6b734829a7d38e30d4471def7abc5b78634"} Nov 02 23:40:39.821850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.821774 4494 scope.go:122] "RemoveContainer" containerID="4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f" Nov 02 23:40:39.858109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.857838 4494 scope.go:122] "RemoveContainer" containerID="39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893" Nov 02 23:40:39.868493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.868465 4494 scope.go:122] "RemoveContainer" containerID="4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f" Nov 02 23:40:39.868791 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:39.868766 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f\": container with ID starting with 4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f not found: ID does not exist" containerID="4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f" Nov 02 23:40:39.868851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.868829 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f"} err="failed to get container status \"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f\": rpc error: code = NotFound desc = could not find container \"4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f\": container with ID starting with 4ee02a2ab5cc6119fcce1f623e8967740dcd66ffa83a8db8294d38c160ad5e6f not found: ID does not exist" Nov 02 23:40:39.868901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.868850 4494 scope.go:122] "RemoveContainer" containerID="39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893" Nov 02 23:40:39.869124 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:39.869106 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893\": container with ID starting with 39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893 not found: ID does not exist" containerID="39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893" Nov 02 23:40:39.869219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:39.869133 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893"} err="failed to get container status \"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893\": rpc error: code = NotFound desc = could not find container \"39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893\": container with ID starting with 39c6f24347619442b6116a292e73442cfeca1c4c604218ee26102e714b6ec893 not found: ID does not exist" Nov 02 23:40:40.025422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.025332 4494 patch_prober.go:29] interesting pod/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Nov 02 23:40:40.025422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.025396 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Nov 02 23:40:40.056474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.056434 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:40:40.089783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.089746 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-94e3ae-openshift-default-957b947cc-84kd9"] Nov 02 23:40:40.092569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.092546 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:40:40.373963 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.373860 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:40.526365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.520141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:40.668668 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:40.668629 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:41.030284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:41.030211 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:40:41.031565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:41.031544 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:40:41.706328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:41.706299 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" path="/var/lib/kubelet/pods/5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff/volumes" Nov 02 23:40:43.428663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:43.428624 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:45.093378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.093336 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" podUID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" containerName="payload-processing" containerID="cri-o://cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7" gracePeriod=25 Nov 02 23:40:45.378753 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.378724 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:40:45.434928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.434893 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\") pod \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " Nov 02 23:40:45.435245 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.435223 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\" (UniqueName: \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\") pod \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\" (UID: \"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4\") " Nov 02 23:40:45.435405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.435384 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume" pod "9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" (UID: "9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:45.435698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.435681 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:45.437392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.437371 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl" pod "9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" (UID: "9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4"). InnerVolumeSpecName "kube-api-access-btgtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:45.537273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.537238 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btgtl\" (UniqueName: \"kubernetes.io/projected/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4-kube-api-access-btgtl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:45.874254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.874210 4494 generic.go:356] "Generic (PLEG): container finished" podID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" containerID="cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7" exitCode=0 Nov 02 23:40:45.874434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.874318 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" Nov 02 23:40:45.874434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.874346 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" event={"ID":"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4","Type":"ContainerDied","Data":"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7"} Nov 02 23:40:45.874434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.874400 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb" event={"ID":"9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4","Type":"ContainerDied","Data":"11c863cc4d6461c2a3e11e08a0f9fe33c03ec560a5c7b693e39b12c63ec5661b"} Nov 02 23:40:45.874434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.874424 4494 scope.go:122] "RemoveContainer" containerID="cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7" Nov 02 23:40:45.885340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.885259 4494 scope.go:122] "RemoveContainer" containerID="cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7" Nov 02 23:40:45.885574 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:45.885551 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7\": container with ID starting with cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7 not found: ID does not exist" containerID="cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7" Nov 02 23:40:45.885638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:45.885581 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7"} err="failed to get container status \"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7\": rpc error: code = NotFound desc = could not find container \"cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7\": container with ID starting with cbfd564a693fc639a6e057f3021597996ffadac4a85c082c7805d65d0c07d3f7 not found: ID does not exist" Nov 02 23:40:46.041136 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:46.041035 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:40:46.045742 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:46.045717 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a5d6d8-697994db4f-jhbxb"] Nov 02 23:40:46.254455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:46.254424 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:47.562707 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:47.562668 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:40:47.699618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:47.699581 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" path="/var/lib/kubelet/pods/9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4/volumes" Nov 02 23:40:48.016161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.016127 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:40:48.016419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.016393 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" podUID="6b93938d-0915-4e2e-a00a-a90b413a245d" containerName="maas-api" containerID="cri-o://020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f" gracePeriod=30 Nov 02 23:40:48.117885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.117846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:48.268572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.268461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:48.343903 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.343871 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:40:48.476021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.475989 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\") pod \"6b93938d-0915-4e2e-a00a-a90b413a245d\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " Nov 02 23:40:48.476221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.476106 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\") pod \"6b93938d-0915-4e2e-a00a-a90b413a245d\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " Nov 02 23:40:48.476221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.476157 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") pod \"6b93938d-0915-4e2e-a00a-a90b413a245d\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " Nov 02 23:40:48.476363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.476313 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\" (UniqueName: \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\") pod \"6b93938d-0915-4e2e-a00a-a90b413a245d\" (UID: \"6b93938d-0915-4e2e-a00a-a90b413a245d\") " Nov 02 23:40:48.476494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.476468 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca" pod "6b93938d-0915-4e2e-a00a-a90b413a245d" (UID: "6b93938d-0915-4e2e-a00a-a90b413a245d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:48.477252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.477216 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93938d-0915-4e2e-a00a-a90b413a245d-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:48.478642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.478614 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls" pod "6b93938d-0915-4e2e-a00a-a90b413a245d" (UID: "6b93938d-0915-4e2e-a00a-a90b413a245d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:48.478897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.478867 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv" pod "6b93938d-0915-4e2e-a00a-a90b413a245d" (UID: "6b93938d-0915-4e2e-a00a-a90b413a245d"). InnerVolumeSpecName "kube-api-access-8sjfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:48.479029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.478875 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs" pod "6b93938d-0915-4e2e-a00a-a90b413a245d" (UID: "6b93938d-0915-4e2e-a00a-a90b413a245d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:40:48.594124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.592287 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:48.594124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.592316 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93938d-0915-4e2e-a00a-a90b413a245d-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:48.594124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.592327 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjfv\" (UniqueName: \"kubernetes.io/projected/6b93938d-0915-4e2e-a00a-a90b413a245d-kube-api-access-8sjfv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:48.897844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.897745 4494 generic.go:356] "Generic (PLEG): container finished" podID="6b93938d-0915-4e2e-a00a-a90b413a245d" containerID="020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f" exitCode=0 Nov 02 23:40:48.897844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.897811 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" Nov 02 23:40:48.898101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.897836 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" event={"ID":"6b93938d-0915-4e2e-a00a-a90b413a245d","Type":"ContainerDied","Data":"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f"} Nov 02 23:40:48.898101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.897898 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd" event={"ID":"6b93938d-0915-4e2e-a00a-a90b413a245d","Type":"ContainerDied","Data":"490e68cdc216257d37693a65c431b5b2cf31c5be742f178d1c004abd2dd1dc0b"} Nov 02 23:40:48.898101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.897919 4494 scope.go:122] "RemoveContainer" containerID="020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f" Nov 02 23:40:48.909308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.909285 4494 scope.go:122] "RemoveContainer" containerID="020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f" Nov 02 23:40:48.909603 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:40:48.909581 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f\": container with ID starting with 020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f not found: ID does not exist" containerID="020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f" Nov 02 23:40:48.909655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:48.909614 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f"} err="failed to get container status \"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f\": rpc error: code = NotFound desc = could not find container \"020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f\": container with ID starting with 020d1dec00b63e0cd3a58e5bc4ab5c05fc2db775bf2d04d6f86dc638ce00e09f not found: ID does not exist" Nov 02 23:40:49.085635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:49.085603 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:40:49.088794 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:49.088772 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a5d6d8-7c88d6df89-427kd"] Nov 02 23:40:49.694225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:49.694189 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b93938d-0915-4e2e-a00a-a90b413a245d" path="/var/lib/kubelet/pods/6b93938d-0915-4e2e-a00a-a90b413a245d/volumes" Nov 02 23:40:49.870682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:49.870649 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:51.838802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:51.838741 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:51.983828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:51.983750 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:53.428681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:53.428651 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:53.962534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:53.962503 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:53.968010 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:53.967987 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:40:53.968224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:53.968204 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" containerID="cri-o://584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a" gracePeriod=30 Nov 02 23:40:55.062374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:55.062337 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:40:57.808874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:57.808837 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:40:59.241522 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.241456 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:40:59.333047 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333010 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333057 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333078 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333213 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333266 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333315 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333358 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333398 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333396 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333433 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333497 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\") pod \"510b384f-58fa-415b-be77-feb8e7eb519c\" (UID: \"510b384f-58fa-415b-be77-feb8e7eb519c\") " Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333500 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:59.333588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333573 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333797 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333960 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333978 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333987 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510b384f-58fa-415b-be77-feb8e7eb519c-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.333996 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.334083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.334005 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.335693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.335664 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "kube-api-access-lpsht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:59.335801 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.335734 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:40:59.336011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.335991 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:40:59.336078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.336063 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy" pod "510b384f-58fa-415b-be77-feb8e7eb519c" (UID: "510b384f-58fa-415b-be77-feb8e7eb519c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:40:59.435104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.435058 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510b384f-58fa-415b-be77-feb8e7eb519c-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.435104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.435101 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsht\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-kube-api-access-lpsht\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.435416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.435117 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510b384f-58fa-415b-be77-feb8e7eb519c-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.435416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.435133 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510b384f-58fa-415b-be77-feb8e7eb519c-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:40:59.991282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.991166 4494 generic.go:356] "Generic (PLEG): container finished" podID="510b384f-58fa-415b-be77-feb8e7eb519c" containerID="584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a" exitCode=0 Nov 02 23:40:59.991282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.991249 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" Nov 02 23:40:59.991496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.991280 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerDied","Data":"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a"} Nov 02 23:40:59.991496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.991345 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb" event={"ID":"510b384f-58fa-415b-be77-feb8e7eb519c","Type":"ContainerDied","Data":"e19bd7cffd032baa34b1325fabeadd079a10aa6d63eb79a0d06080ef78064acb"} Nov 02 23:40:59.991496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:40:59.991368 4494 scope.go:122] "RemoveContainer" containerID="584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a" Nov 02 23:41:00.002690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.002664 4494 scope.go:122] "RemoveContainer" containerID="0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf" Nov 02 23:41:00.012088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.012068 4494 scope.go:122] "RemoveContainer" containerID="584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a" Nov 02 23:41:00.012390 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:41:00.012368 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a\": container with ID starting with 584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a not found: ID does not exist" containerID="584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a" Nov 02 23:41:00.012456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.012399 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a"} err="failed to get container status \"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a\": rpc error: code = NotFound desc = could not find container \"584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a\": container with ID starting with 584e00e49a7244d93a53745430cc5e3bd4e78bd96e7ec9ba612786f4ecb9824a not found: ID does not exist" Nov 02 23:41:00.012456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.012420 4494 scope.go:122] "RemoveContainer" containerID="0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf" Nov 02 23:41:00.012647 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:41:00.012623 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf\": container with ID starting with 0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf not found: ID does not exist" containerID="0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf" Nov 02 23:41:00.012695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.012654 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf"} err="failed to get container status \"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf\": rpc error: code = NotFound desc = could not find container \"0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf\": container with ID starting with 0e9a27dd273e5e33792faa472b80e687aa5fb418b9c38234546b55fc376783cf not found: ID does not exist" Nov 02 23:41:00.159393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.159362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:00.222000 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.221964 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:41:00.225506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:00.225483 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a5d6d8-openshift-default-78657b9c45-2njbb"] Nov 02 23:41:01.694641 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:01.694609 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" path="/var/lib/kubelet/pods/510b384f-58fa-415b-be77-feb8e7eb519c/volumes" Nov 02 23:41:05.048212 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:05.048152 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:09.946191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:09.946147 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:13.544983 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:13.544942 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:13.803890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:13.803780 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:14.459628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:14.458096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:14.775148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:14.775014 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:14.966861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:14.966828 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:17.560840 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:41:17.560582 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:41:18.310439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:18.310408 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:18.477328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:18.477292 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:18.630698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:18.630615 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:21.769624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:21.769584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:21.921961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:21.921915 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:22.307155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307117 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:41:22.307787 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307770 4494 cpu_manager.go:414] "containerMap: removing container" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307788 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307804 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" containerName="payload-processing" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307809 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" containerName="payload-processing" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307821 4494 cpu_manager.go:414] "containerMap: removing container" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307825 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307837 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307844 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307872 4494 cpu_manager.go:414] "containerMap: removing container" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" Nov 02 23:41:22.307878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307880 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307895 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6b93938d-0915-4e2e-a00a-a90b413a245d" containerName="maas-api" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307902 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b93938d-0915-4e2e-a00a-a90b413a245d" containerName="maas-api" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307915 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.307922 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308065 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="668c3b82-4242-43eb-946f-78a802c34d4e" containerName="payload-pre-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308081 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b93938d-0915-4e2e-a00a-a90b413a245d" containerName="maas-api" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308091 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d96b446-1ec0-45a7-9358-a5d9e31ee69e" containerName="payload-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308101 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308113 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308121 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbbbcd4-fd16-4f1d-b699-f00d6a3c79c4" containerName="payload-processing" Nov 02 23:41:22.308345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.308129 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="510b384f-58fa-415b-be77-feb8e7eb519c" containerName="istio-proxy" Nov 02 23:41:22.313343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.313324 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.368697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.368656 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:41:22.376964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.376930 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.377264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.377243 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgx8b\" (UniqueName: \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.414936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.414911 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-875049\"" Nov 02 23:41:22.415094 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.415050 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-875049-dockercfg-k646k\"" Nov 02 23:41:22.478278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.478243 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgx8b\" (UniqueName: \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.479025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.478364 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.480205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.480158 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.541319 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.541287 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgx8b\" (UniqueName: \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\") pod \"payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:22.626998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:22.626906 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:23.011724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:23.011691 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:41:23.012431 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:41:23.012407 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a56e2ad_6d75_4520_aab0_fa77d6c9744b.slice/crio-9445ba28c096511480f6f8dfc37acd8910d1df159751de921aa46ebf84331d94 WatchSource:0}: Error finding container 9445ba28c096511480f6f8dfc37acd8910d1df159751de921aa46ebf84331d94: Status 404 returned error can't find the container with id 9445ba28c096511480f6f8dfc37acd8910d1df159751de921aa46ebf84331d94 Nov 02 23:41:23.158539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:23.158506 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" event={"ID":"7a56e2ad-6d75-4520-aab0-fa77d6c9744b","Type":"ContainerStarted","Data":"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5"} Nov 02 23:41:23.158719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:23.158562 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" event={"ID":"7a56e2ad-6d75-4520-aab0-fa77d6c9744b","Type":"ContainerStarted","Data":"9445ba28c096511480f6f8dfc37acd8910d1df159751de921aa46ebf84331d94"} Nov 02 23:41:23.158719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:23.158590 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:23.277195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:23.277071 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" podStartSLOduration=1.277056062 podStartE2EDuration="1.277056062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:23.27469157 +0000 UTC m=+3197.704561430" watchObservedRunningTime="2026-11-02 23:41:23.277056062 +0000 UTC m=+3197.706925922" Nov 02 23:41:25.338339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:25.338284 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:27.335905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:27.335863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:30.136564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:30.136530 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:30.563766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:30.563731 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:34.168128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:34.168098 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:41:35.397380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:35.397346 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:35.998399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:35.998364 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:37.122967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:37.122933 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:37.266403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:37.266370 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:38.762923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:38.762877 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:41:40.653869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:40.653831 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:41.743051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:41.743014 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:45.545459 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:45.545423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:47.562782 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:41:47.562744 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:41:50.445971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:50.445902 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:41:55.348523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:41:55.348484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:00.292114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:00.292055 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:05.131331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:05.131258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:09.960758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:09.960675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:13.889603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:13.889556 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:15.412681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:15.412643 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:17.561549 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:17.561511 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:42:18.004541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:18.004508 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:42:19.110264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.110227 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:42:19.110692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.110462 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" podUID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" containerName="maas-api" containerID="cri-o://95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55" gracePeriod=30 Nov 02 23:42:19.209969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.209934 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:19.427206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.427156 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:42:19.544153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.544104 4494 generic.go:356] "Generic (PLEG): container finished" podID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" containerID="95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55" exitCode=0 Nov 02 23:42:19.544153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.544154 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" Nov 02 23:42:19.544420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.544160 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" event={"ID":"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7","Type":"ContainerDied","Data":"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55"} Nov 02 23:42:19.544420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.544274 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r" event={"ID":"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7","Type":"ContainerDied","Data":"5a664c0235bfe62ccf9d92f9b0ae0eb2629b5503095e2dc732008314f4b18c00"} Nov 02 23:42:19.544420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.544311 4494 scope.go:122] "RemoveContainer" containerID="95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55" Nov 02 23:42:19.554577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.554545 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\") pod \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " Nov 02 23:42:19.554725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.554631 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\") pod \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " Nov 02 23:42:19.554793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.554777 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") pod \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " Nov 02 23:42:19.554844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.554830 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\" (UniqueName: \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\") pod \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\" (UID: \"c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7\") " Nov 02 23:42:19.558259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.555871 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca" pod "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" (UID: "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:19.558259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.557974 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs" pod "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" (UID: "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:42:19.558259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.558228 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf" pod "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" (UID: "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7"). InnerVolumeSpecName "kube-api-access-pkqbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:19.558912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.558894 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls" pod "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" (UID: "c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:42:19.566200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.566162 4494 scope.go:122] "RemoveContainer" containerID="95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55" Nov 02 23:42:19.566589 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:19.566566 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55\": container with ID starting with 95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55 not found: ID does not exist" containerID="95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55" Nov 02 23:42:19.566689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.566597 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55"} err="failed to get container status \"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55\": rpc error: code = NotFound desc = could not find container \"95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55\": container with ID starting with 95db724459077d921861a4bf4a8f1335596532ca10f6b9fde6e9c0e31452dc55 not found: ID does not exist" Nov 02 23:42:19.656330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.656229 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:19.656330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.656266 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:19.656330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.656281 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqbf\" (UniqueName: \"kubernetes.io/projected/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-kube-api-access-pkqbf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:19.656330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.656299 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:19.963042 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:19.963008 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:42:20.013500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:20.013470 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cdec0-5c9c8fcc55-jll4r"] Nov 02 23:42:20.976022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:20.975969 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:21.693919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:21.693885 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" path="/var/lib/kubelet/pods/c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7/volumes" Nov 02 23:42:22.935134 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:22.935102 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:23.082100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:23.082059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:23.246457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:23.246365 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:24.493858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:24.493818 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:24.969856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:24.969826 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:42:24.970088 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:24.970045 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" containerID="cri-o://175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895" gracePeriod=30 Nov 02 23:42:24.981582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:24.981557 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:25.784439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.784405 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:42:25.785095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785080 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" containerName="maas-api" Nov 02 23:42:25.785147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785097 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" containerName="maas-api" Nov 02 23:42:25.785147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785124 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:42:25.785147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785129 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:42:25.785271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785259 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d7b3fa-74e4-4f0e-ab91-0ce84651d2a7" containerName="maas-api" Nov 02 23:42:25.785308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.785285 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f03dff1-7a1b-4b65-8f9f-00443ed3a2ff" containerName="istio-proxy" Nov 02 23:42:25.789965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.789941 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:25.820882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.820852 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:25.843426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.843393 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-875049-serving-cert\"" Nov 02 23:42:25.847114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.847085 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:42:25.949271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.949239 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:25.949448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.949367 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:25.949448 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.949438 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:25.949564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:25.949505 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv7j\" (UniqueName: \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.050417 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.050329 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:26.050902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.050863 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.051065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.050969 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.051065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.051060 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.051240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.051110 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv7j\" (UniqueName: \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.052211 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.052186 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.067156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.067092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.068819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.068779 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.113102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.113064 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv7j\" (UniqueName: \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\") pod \"maas-api-e2e-autores-875049-6675dc6685-wwpd9\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.201297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.201242 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:26.404680 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.404597 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:26.782614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:26.782586 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:42:26.783724 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:42:26.783694 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680e76ef_c2ea_48e2_8bf0_4b176eea4176.slice/crio-0253cc879c869a324dcd09dd68c631bd01ca204ca6189c1cc31b3e145baa66e0 WatchSource:0}: Error finding container 0253cc879c869a324dcd09dd68c631bd01ca204ca6189c1cc31b3e145baa66e0: Status 404 returned error can't find the container with id 0253cc879c869a324dcd09dd68c631bd01ca204ca6189c1cc31b3e145baa66e0 Nov 02 23:42:27.602643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:27.602613 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" event={"ID":"680e76ef-c2ea-48e2-8bf0-4b176eea4176","Type":"ContainerStarted","Data":"0253cc879c869a324dcd09dd68c631bd01ca204ca6189c1cc31b3e145baa66e0"} Nov 02 23:42:28.610057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:28.610021 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" event={"ID":"680e76ef-c2ea-48e2-8bf0-4b176eea4176","Type":"ContainerStarted","Data":"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493"} Nov 02 23:42:28.610481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:28.610080 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:28.730956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:28.730907 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" podStartSLOduration=2.783682799 podStartE2EDuration="3.730890796s" totalImagesPullingTime="947.207997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:42:28.728499576 +0000 UTC m=+3263.158369444" watchObservedRunningTime="2026-11-02 23:42:28.730890796 +0000 UTC m=+3263.160760654" Nov 02 23:42:28.896138 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:28.896052 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:42:28.896316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:28.896300 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" containerID="cri-o://8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f" gracePeriod=30 Nov 02 23:42:30.232209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.232165 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:42:30.309880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.309793 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.309880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.309839 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.309880 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.309875 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.309894 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.309974 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310016 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310055 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310126 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310160 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\") pod \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\" (UID: \"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0\") " Nov 02 23:42:30.310640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310297 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:30.310640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310309 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:30.310640 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310451 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310636 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310659 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310873 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310898 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310913 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310927 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.311049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.310941 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.312567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.312538 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:30.312732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.312713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "kube-api-access-5m59w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:30.312783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.312741 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:30.312783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.312759 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo" pod "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" (UID: "30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:42:30.411854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.411823 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.411854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.411849 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.411854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.411858 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.412080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.411874 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m59w\" (UniqueName: \"kubernetes.io/projected/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0-kube-api-access-5m59w\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:30.627331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.627246 4494 generic.go:356] "Generic (PLEG): container finished" podID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerID="175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895" exitCode=0 Nov 02 23:42:30.627331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.627296 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" Nov 02 23:42:30.627577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.627336 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerDied","Data":"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895"} Nov 02 23:42:30.627577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.627401 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565" event={"ID":"30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0","Type":"ContainerDied","Data":"b130c4cd860c699913bd8ccfd6d4bdc46895dd93e2117a579a6215e489142810"} Nov 02 23:42:30.627577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.627431 4494 scope.go:122] "RemoveContainer" containerID="175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895" Nov 02 23:42:30.639104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.639082 4494 scope.go:122] "RemoveContainer" containerID="d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9" Nov 02 23:42:30.648221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.648203 4494 scope.go:122] "RemoveContainer" containerID="175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895" Nov 02 23:42:30.648553 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:30.648518 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895\": container with ID starting with 175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895 not found: ID does not exist" containerID="175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895" Nov 02 23:42:30.648649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.648563 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895"} err="failed to get container status \"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895\": rpc error: code = NotFound desc = could not find container \"175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895\": container with ID starting with 175f7adfc5d0d712719a1d97d28f5a2047988a6def68d398f1158439d6c1f895 not found: ID does not exist" Nov 02 23:42:30.648649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.648588 4494 scope.go:122] "RemoveContainer" containerID="d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9" Nov 02 23:42:30.648871 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:30.648852 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9\": container with ID starting with d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9 not found: ID does not exist" containerID="d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9" Nov 02 23:42:30.648916 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.648879 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9"} err="failed to get container status \"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9\": rpc error: code = NotFound desc = could not find container \"d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9\": container with ID starting with d732de24e2899d62412f8c9e97fc22773f4c876359930ca1efc779167f7aa2f9 not found: ID does not exist" Nov 02 23:42:30.795925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.795895 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:42:30.799541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:30.799516 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4cdec0-openshift-default-7d896d78f-sq565"] Nov 02 23:42:31.264758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:31.264721 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:31.694877 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:31.694843 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" path="/var/lib/kubelet/pods/30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0/volumes" Nov 02 23:42:32.355215 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:32.355179 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:33.086392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:33.086360 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:42:33.198264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:33.198226 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:42:33.449060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:33.449026 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:33.596139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:33.596112 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:34.178582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.178515 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:42:34.256969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.256935 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257001 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257047 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257068 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257130 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257154 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257247 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257274 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:34.257497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257314 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257354 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\") pod \"456a86bd-4db5-43f7-aea4-88e906b845fe\" (UID: \"456a86bd-4db5-43f7-aea4-88e906b845fe\") " Nov 02 23:42:34.257497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257464 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:34.257775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257558 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:34.257775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257604 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:34.257775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257635 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:34.257946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257868 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.257946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257887 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.257946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257903 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.257946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257917 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.257946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.257942 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456a86bd-4db5-43f7-aea4-88e906b845fe-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.259557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.259529 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:42:34.259784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.259767 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:42:34.260034 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.260015 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:34.260102 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.260044 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv" pod "456a86bd-4db5-43f7-aea4-88e906b845fe" (UID: "456a86bd-4db5-43f7-aea4-88e906b845fe"). InnerVolumeSpecName "kube-api-access-bpxhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:34.359523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.359490 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.359523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.359518 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.359523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.359528 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456a86bd-4db5-43f7-aea4-88e906b845fe-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.359783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.359537 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxhv\" (UniqueName: \"kubernetes.io/projected/456a86bd-4db5-43f7-aea4-88e906b845fe-kube-api-access-bpxhv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:34.626400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.626325 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:42:34.659021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.658991 4494 generic.go:356] "Generic (PLEG): container finished" podID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerID="8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f" exitCode=0 Nov 02 23:42:34.659226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.659057 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" Nov 02 23:42:34.659226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.659077 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" event={"ID":"456a86bd-4db5-43f7-aea4-88e906b845fe","Type":"ContainerDied","Data":"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f"} Nov 02 23:42:34.659226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.659135 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h" event={"ID":"456a86bd-4db5-43f7-aea4-88e906b845fe","Type":"ContainerDied","Data":"93c43c9305cd707f87fa70a51060c981855ebb2f3989fb92644d5b45358a9fdb"} Nov 02 23:42:34.659226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.659166 4494 scope.go:122] "RemoveContainer" containerID="8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f" Nov 02 23:42:34.670642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.670616 4494 scope.go:122] "RemoveContainer" containerID="8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f" Nov 02 23:42:34.670964 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:34.670943 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f\": container with ID starting with 8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f not found: ID does not exist" containerID="8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f" Nov 02 23:42:34.671045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.670977 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f"} err="failed to get container status \"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f\": rpc error: code = NotFound desc = could not find container \"8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f\": container with ID starting with 8429141281f783286346e94f3088c0690447a26be045b7102b6c2531fde8335f not found: ID does not exist" Nov 02 23:42:34.877277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.864832 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:42:34.881205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:34.880932 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-08c2c6-openshift-default-7489bf87c7-nl52h"] Nov 02 23:42:35.694644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:35.694610 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" path="/var/lib/kubelet/pods/456a86bd-4db5-43f7-aea4-88e906b845fe/volumes" Nov 02 23:42:36.355611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:36.355578 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:38.087625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.087582 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" podUID="f1636881-54c7-4da7-b4f3-c4330cffac58" containerName="payload-processing" containerID="cri-o://787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3" gracePeriod=25 Nov 02 23:42:38.198945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.198901 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" podUID="6117a085-64e6-4ecf-a781-ba89faba6cc0" containerName="payload-pre-processing" containerID="cri-o://2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04" gracePeriod=25 Nov 02 23:42:38.212107 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.212058 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:38.356497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.356429 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:42:38.425072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.425035 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\") pod \"f1636881-54c7-4da7-b4f3-c4330cffac58\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " Nov 02 23:42:38.425308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.425197 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\" (UniqueName: \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\") pod \"f1636881-54c7-4da7-b4f3-c4330cffac58\" (UID: \"f1636881-54c7-4da7-b4f3-c4330cffac58\") " Nov 02 23:42:38.425496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.425473 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume" pod "f1636881-54c7-4da7-b4f3-c4330cffac58" (UID: "f1636881-54c7-4da7-b4f3-c4330cffac58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:38.425802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.425779 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1636881-54c7-4da7-b4f3-c4330cffac58-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:38.428146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.428117 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b" pod "f1636881-54c7-4da7-b4f3-c4330cffac58" (UID: "f1636881-54c7-4da7-b4f3-c4330cffac58"). InnerVolumeSpecName "kube-api-access-zgq4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:38.451878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.451860 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:42:38.526356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.526326 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\") pod \"6117a085-64e6-4ecf-a781-ba89faba6cc0\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " Nov 02 23:42:38.526533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.526397 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\" (UniqueName: \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\") pod \"6117a085-64e6-4ecf-a781-ba89faba6cc0\" (UID: \"6117a085-64e6-4ecf-a781-ba89faba6cc0\") " Nov 02 23:42:38.526706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.526685 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume" pod "6117a085-64e6-4ecf-a781-ba89faba6cc0" (UID: "6117a085-64e6-4ecf-a781-ba89faba6cc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:42:38.526848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.526834 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgq4b\" (UniqueName: \"kubernetes.io/projected/f1636881-54c7-4da7-b4f3-c4330cffac58-kube-api-access-zgq4b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:38.526886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.526851 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6117a085-64e6-4ecf-a781-ba89faba6cc0-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:38.528728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.528707 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z" pod "6117a085-64e6-4ecf-a781-ba89faba6cc0" (UID: "6117a085-64e6-4ecf-a781-ba89faba6cc0"). InnerVolumeSpecName "kube-api-access-hnt7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:42:38.628610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.628528 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnt7z\" (UniqueName: \"kubernetes.io/projected/6117a085-64e6-4ecf-a781-ba89faba6cc0-kube-api-access-hnt7z\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:42:38.687808 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.687678 4494 generic.go:356] "Generic (PLEG): container finished" podID="6117a085-64e6-4ecf-a781-ba89faba6cc0" containerID="2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04" exitCode=0 Nov 02 23:42:38.688300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.687728 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" Nov 02 23:42:38.688438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.687851 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" event={"ID":"6117a085-64e6-4ecf-a781-ba89faba6cc0","Type":"ContainerDied","Data":"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04"} Nov 02 23:42:38.688438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.688429 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66" event={"ID":"6117a085-64e6-4ecf-a781-ba89faba6cc0","Type":"ContainerDied","Data":"aaf56ce339c1036dc730e724f8c240a70b7f86be5d77cb9ee9c450238d42ec12"} Nov 02 23:42:38.688588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.688466 4494 scope.go:122] "RemoveContainer" containerID="2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04" Nov 02 23:42:38.689453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.689365 4494 generic.go:356] "Generic (PLEG): container finished" podID="f1636881-54c7-4da7-b4f3-c4330cffac58" containerID="787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3" exitCode=0 Nov 02 23:42:38.689453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.689393 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" event={"ID":"f1636881-54c7-4da7-b4f3-c4330cffac58","Type":"ContainerDied","Data":"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3"} Nov 02 23:42:38.689453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.689406 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" Nov 02 23:42:38.689453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.689438 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw" event={"ID":"f1636881-54c7-4da7-b4f3-c4330cffac58","Type":"ContainerDied","Data":"c4421dfcd5cfb762f4e83ad7cebd128814f847cf674d2b1895f845c00331cef5"} Nov 02 23:42:38.698893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.698878 4494 scope.go:122] "RemoveContainer" containerID="2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04" Nov 02 23:42:38.699203 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:38.699166 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04\": container with ID starting with 2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04 not found: ID does not exist" containerID="2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04" Nov 02 23:42:38.699353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.699210 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04"} err="failed to get container status \"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04\": rpc error: code = NotFound desc = could not find container \"2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04\": container with ID starting with 2f92de08951639813d3c77201847ea266d618fc792d195060247992191d4ab04 not found: ID does not exist" Nov 02 23:42:38.699353 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.699235 4494 scope.go:122] "RemoveContainer" containerID="787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3" Nov 02 23:42:38.709274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.709250 4494 scope.go:122] "RemoveContainer" containerID="787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3" Nov 02 23:42:38.709528 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:38.709509 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3\": container with ID starting with 787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3 not found: ID does not exist" containerID="787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3" Nov 02 23:42:38.709590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.709533 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3"} err="failed to get container status \"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3\": rpc error: code = NotFound desc = could not find container \"787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3\": container with ID starting with 787feaccc3276879b7c38460c0b6d66e788792b03c1ad5cccacccf9419ffa0a3 not found: ID does not exist" Nov 02 23:42:38.854760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.854678 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:42:38.858072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.858050 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dbe7b2-bbfb69495-pnf66"] Nov 02 23:42:38.968295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.968265 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:42:38.973895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:38.973865 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbe7b2-d9fc4c97c-cbfxw"] Nov 02 23:42:39.695405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:39.695370 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6117a085-64e6-4ecf-a781-ba89faba6cc0" path="/var/lib/kubelet/pods/6117a085-64e6-4ecf-a781-ba89faba6cc0/volumes" Nov 02 23:42:39.695968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:39.695947 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1636881-54c7-4da7-b4f3-c4330cffac58" path="/var/lib/kubelet/pods/f1636881-54c7-4da7-b4f3-c4330cffac58/volumes" Nov 02 23:42:41.250509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:41.250472 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:44.310621 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:44.310582 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:46.255080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:46.255042 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:47.562970 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:42:47.562876 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:42:51.170008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:51.169958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:42:56.047158 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:42:56.047119 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:01.025187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:01.025143 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:05.838891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:05.838854 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:10.791280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:10.791247 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:14.265163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:14.265126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:14.420221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:14.420154 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:14.703302 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:14.703267 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:43:15.811759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:15.811729 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:17.561025 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:43:17.560989 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:43:20.843704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:20.843666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:25.745791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:25.742358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:30.559926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:30.559884 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:35.405057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:35.405023 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:39.598738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:39.598695 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:43:39.599638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:39.599606 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" podUID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" containerName="maas-api" containerID="cri-o://2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984" gracePeriod=30 Nov 02 23:43:39.686852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:39.686823 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:39.904956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:39.904922 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:43:40.010096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010057 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\") pod \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " Nov 02 23:43:40.010322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010156 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\") pod \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " Nov 02 23:43:40.010322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010246 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") pod \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " Nov 02 23:43:40.010322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010279 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\" (UniqueName: \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\") pod \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\" (UID: \"4cefe6d3-ef25-44ca-89d6-cecc54a0519b\") " Nov 02 23:43:40.010547 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010525 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca" pod "4cefe6d3-ef25-44ca-89d6-cecc54a0519b" (UID: "4cefe6d3-ef25-44ca-89d6-cecc54a0519b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:40.010749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.010735 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:40.012350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.012323 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs" pod "4cefe6d3-ef25-44ca-89d6-cecc54a0519b" (UID: "4cefe6d3-ef25-44ca-89d6-cecc54a0519b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:43:40.012469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.012397 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls" pod "4cefe6d3-ef25-44ca-89d6-cecc54a0519b" (UID: "4cefe6d3-ef25-44ca-89d6-cecc54a0519b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:43:40.012469 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.012454 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd" pod "4cefe6d3-ef25-44ca-89d6-cecc54a0519b" (UID: "4cefe6d3-ef25-44ca-89d6-cecc54a0519b"). InnerVolumeSpecName "kube-api-access-gc9rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:40.078961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.078916 4494 generic.go:356] "Generic (PLEG): container finished" podID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" containerID="2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984" exitCode=0 Nov 02 23:43:40.078961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.078961 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" Nov 02 23:43:40.079278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.078990 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" event={"ID":"4cefe6d3-ef25-44ca-89d6-cecc54a0519b","Type":"ContainerDied","Data":"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984"} Nov 02 23:43:40.079278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.079062 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2" event={"ID":"4cefe6d3-ef25-44ca-89d6-cecc54a0519b","Type":"ContainerDied","Data":"51596a506a86ebd6643c62ef5a2535492bb54f3ea798f8402e89a818ce073922"} Nov 02 23:43:40.079278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.079085 4494 scope.go:122] "RemoveContainer" containerID="2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984" Nov 02 23:43:40.090101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.090082 4494 scope.go:122] "RemoveContainer" containerID="2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984" Nov 02 23:43:40.090431 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:43:40.090411 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984\": container with ID starting with 2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984 not found: ID does not exist" containerID="2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984" Nov 02 23:43:40.090499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.090439 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984"} err="failed to get container status \"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984\": rpc error: code = NotFound desc = could not find container \"2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984\": container with ID starting with 2ef0834ba2a134ef17a95241439632b889ff903d93c6273597238a6a63a32984 not found: ID does not exist" Nov 02 23:43:40.111804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.111779 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:40.111891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.111805 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc9rd\" (UniqueName: \"kubernetes.io/projected/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-kube-api-access-gc9rd\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:40.111891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.111816 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cefe6d3-ef25-44ca-89d6-cecc54a0519b-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:40.246451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.246378 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:43:40.249743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:40.249712 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbe7b2-76699f6d4c-4wnb2"] Nov 02 23:43:41.383111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:41.383069 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:41.549610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:41.549569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:41.693873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:41.693842 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" path="/var/lib/kubelet/pods/4cefe6d3-ef25-44ca-89d6-cecc54a0519b/volumes" Nov 02 23:43:42.816056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:42.816019 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:42.962220 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:42.962165 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:44.548888 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:44.548857 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:45.982139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:45.982104 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:46.128894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:46.128858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:46.700777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:46.700747 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:47.575253 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:43:47.562734 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:43:47.604562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:47.604533 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:48.089786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:48.089754 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:48.095230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:48.095206 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:43:48.095462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:48.095424 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" containerID="cri-o://cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5" gracePeriod=30 Nov 02 23:43:48.900157 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:48.900122 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:49.630711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.630682 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 02 23:43:49.631327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631309 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6117a085-64e6-4ecf-a781-ba89faba6cc0" containerName="payload-pre-processing" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631329 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6117a085-64e6-4ecf-a781-ba89faba6cc0" containerName="payload-pre-processing" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631343 4494 cpu_manager.go:414] "containerMap: removing container" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631349 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631358 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f1636881-54c7-4da7-b4f3-c4330cffac58" containerName="payload-processing" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631363 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1636881-54c7-4da7-b4f3-c4330cffac58" containerName="payload-processing" Nov 02 23:43:49.631376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631376 4494 cpu_manager.go:414] "containerMap: removing container" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631381 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631398 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" containerName="maas-api" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631403 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" containerName="maas-api" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631421 4494 cpu_manager.go:414] "containerMap: removing container" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631426 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631535 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6117a085-64e6-4ecf-a781-ba89faba6cc0" containerName="payload-pre-processing" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631544 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="456a86bd-4db5-43f7-aea4-88e906b845fe" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631549 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1636881-54c7-4da7-b4f3-c4330cffac58" containerName="payload-processing" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631555 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631562 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cefe6d3-ef25-44ca-89d6-cecc54a0519b" containerName="maas-api" Nov 02 23:43:49.631578 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.631572 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c17f1d-cfc3-48b5-9c9e-9acdcfab03f0" containerName="istio-proxy" Nov 02 23:43:49.636606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.636588 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.719412 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.719383 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:49.723258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723227 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723368 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffr5\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723404 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723555 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723429 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723570 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723659 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723704 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723789 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.723908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.723892 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.736009 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.735984 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 02 23:43:49.742821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.742802 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-882a2a-openshift-default-dockercfg-vlmq6\"" Nov 02 23:43:49.825682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825637 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.825864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825716 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffr5\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.825864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825744 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.825864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825761 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.825864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825805 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.825864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825849 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825883 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825911 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.825979 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.826255 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.826410 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.826443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.826424 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.827104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.827075 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.827280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.827150 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.828629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.828602 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.828975 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.828954 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.887275 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.887211 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.887506 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.887484 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffr5\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\") pod \"e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.949763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.949731 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:49.986565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:49.986535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:50.399476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:50.399452 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 02 23:43:50.400627 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:43:50.400600 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3640416e_01aa_4410_b91e_8937ad196b86.slice/crio-206d49d7fee50089442640378080ed7aff73c345a794927b3a5534b0de6e143f WatchSource:0}: Error finding container 206d49d7fee50089442640378080ed7aff73c345a794927b3a5534b0de6e143f: Status 404 returned error can't find the container with id 206d49d7fee50089442640378080ed7aff73c345a794927b3a5534b0de6e143f Nov 02 23:43:50.403363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:50.403331 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:50.403477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:50.403409 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:50.403477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:50.403452 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:43:51.158447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:51.158413 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerStarted","Data":"2587eb675d601428f2bf03c6e13894d0f4ef2851be554a690cbb88fdcef15d9e"} Nov 02 23:43:51.158870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:51.158471 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerStarted","Data":"206d49d7fee50089442640378080ed7aff73c345a794927b3a5534b0de6e143f"} Nov 02 23:43:51.365828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:51.365778 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podStartSLOduration=2.3657612009999998 podStartE2EDuration="2.365761201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:51.330091926 +0000 UTC m=+3345.759961788" watchObservedRunningTime="2026-11-02 23:43:51.365761201 +0000 UTC m=+3345.795631061" Nov 02 23:43:51.950553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:51.950519 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:51.957907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:51.957877 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:52.083750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:52.083676 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:52.165100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:52.165070 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:52.165926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:52.165906 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:43:53.369883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.369821 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:43:53.489551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489521 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489557 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489717 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489769 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489797 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489828 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489859 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489918 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.489995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.489963 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\") pod \"84b5f827-5dab-4336-96f3-9279ce9f60bd\" (UID: \"84b5f827-5dab-4336-96f3-9279ce9f60bd\") " Nov 02 23:43:53.490210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490018 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:53.490210 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490054 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:53.490332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490272 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:53.490332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490282 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:53.490439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490338 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:53.490580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490561 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.490666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490584 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b5f827-5dab-4336-96f3-9279ce9f60bd-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.490666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490612 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.490666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490627 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.490666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.490640 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.492367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.492340 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "kube-api-access-lh8qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:53.492367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.492350 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:53.492817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.492794 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:53.492817 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.492798 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy" pod "84b5f827-5dab-4336-96f3-9279ce9f60bd" (UID: "84b5f827-5dab-4336-96f3-9279ce9f60bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:53.592243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.592209 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.592243 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.592241 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.592457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.592253 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b5f827-5dab-4336-96f3-9279ce9f60bd-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:53.592457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:53.592265 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh8qn\" (UniqueName: \"kubernetes.io/projected/84b5f827-5dab-4336-96f3-9279ce9f60bd-kube-api-access-lh8qn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:43:54.146700 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.146595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:54.178914 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.178883 4494 generic.go:356] "Generic (PLEG): container finished" podID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerID="cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5" exitCode=0 Nov 02 23:43:54.179143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.178956 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" Nov 02 23:43:54.179143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.178982 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerDied","Data":"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5"} Nov 02 23:43:54.179143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.179037 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr" event={"ID":"84b5f827-5dab-4336-96f3-9279ce9f60bd","Type":"ContainerDied","Data":"45615cbfd840d97618e84bc581f2a8828120eaf5e3897e9ae37bd972507b8d54"} Nov 02 23:43:54.179143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.179062 4494 scope.go:122] "RemoveContainer" containerID="cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5" Nov 02 23:43:54.192394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.192373 4494 scope.go:122] "RemoveContainer" containerID="0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386" Nov 02 23:43:54.201643 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.201626 4494 scope.go:122] "RemoveContainer" containerID="cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5" Nov 02 23:43:54.201900 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:43:54.201885 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5\": container with ID starting with cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5 not found: ID does not exist" containerID="cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5" Nov 02 23:43:54.201956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.201912 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5"} err="failed to get container status \"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5\": rpc error: code = NotFound desc = could not find container \"cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5\": container with ID starting with cf65257d1bc7c2331d7d079b689500b5705d82429d1cc33f5498413cc4e555e5 not found: ID does not exist" Nov 02 23:43:54.201956 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.201944 4494 scope.go:122] "RemoveContainer" containerID="0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386" Nov 02 23:43:54.202224 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:43:54.202203 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386\": container with ID starting with 0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386 not found: ID does not exist" containerID="0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386" Nov 02 23:43:54.202278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.202229 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386"} err="failed to get container status \"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386\": rpc error: code = NotFound desc = could not find container \"0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386\": container with ID starting with 0c44451a93ae32e29a70a61e5b73e5d2ff2b228777412fad1cfb50b145215386 not found: ID does not exist" Nov 02 23:43:54.311629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.311593 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:54.348097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.348066 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:43:54.398366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.398289 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbe7b2-openshift-default-5cd9cbd769-7jnwr"] Nov 02 23:43:54.459252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:54.459216 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:55.694840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:55.694806 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" path="/var/lib/kubelet/pods/84b5f827-5dab-4336-96f3-9279ce9f60bd/volumes" Nov 02 23:43:57.324923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.324882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:57.476958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.476924 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:57.640278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.631859 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:43:57.736310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.736279 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 02 23:43:57.736890 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.736876 4494 cpu_manager.go:414] "containerMap: removing container" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.736941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.736892 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.737019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.737010 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.742084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.742064 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:57.799299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.799261 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 02 23:43:57.847198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.846611 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-882a2a-dockercfg-xh89s\"" Nov 02 23:43:57.847198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.846786 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-882a2a\"" Nov 02 23:43:57.857687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.849977 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:57.883713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.881498 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpm9\" (UniqueName: \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:57.904247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.902458 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 02 23:43:57.904247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.903330 4494 cpu_manager.go:414] "containerMap: removing container" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.904247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.903347 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.904247 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.903455 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b5f827-5dab-4336-96f3-9279ce9f60bd" containerName="istio-proxy" Nov 02 23:43:57.908268 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.908235 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:57.951659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.951626 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 02 23:43:57.983113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.983076 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpm9\" (UniqueName: \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:57.983377 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.983355 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:57.984005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:57.983990 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:58.046085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.046056 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpm9\" (UniqueName: \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\") pod \"payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:58.057123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.057098 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:58.084360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.084333 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.084576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.084379 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2srg\" (UniqueName: \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.185501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.185464 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.185688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.185516 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2srg\" (UniqueName: \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.186166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.186146 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.247848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.247811 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2srg\" (UniqueName: \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\") pod \"payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.441440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.441408 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 02 23:43:58.442681 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:43:58.442654 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091c3f43_8afb_40cb_978c_fffd8b2bd867.slice/crio-eb276ec1afd22b01787e2ee7dea15ba4c52e0d2ec8bd85eafff6d0f857b7b96e WatchSource:0}: Error finding container eb276ec1afd22b01787e2ee7dea15ba4c52e0d2ec8bd85eafff6d0f857b7b96e: Status 404 returned error can't find the container with id eb276ec1afd22b01787e2ee7dea15ba4c52e0d2ec8bd85eafff6d0f857b7b96e Nov 02 23:43:58.522867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.522840 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:58.905308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:58.905281 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 02 23:43:58.906344 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:43:58.906318 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7815790_5819_412d_a30d_dd0541587c3b.slice/crio-09bbe2ae73a4eaca55ea46ec76ccd63170efe65135dcdf76a042d546ec789931 WatchSource:0}: Error finding container 09bbe2ae73a4eaca55ea46ec76ccd63170efe65135dcdf76a042d546ec789931: Status 404 returned error can't find the container with id 09bbe2ae73a4eaca55ea46ec76ccd63170efe65135dcdf76a042d546ec789931 Nov 02 23:43:59.212148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.212114 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" event={"ID":"091c3f43-8afb-40cb-978c-fffd8b2bd867","Type":"ContainerStarted","Data":"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88"} Nov 02 23:43:59.212404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.212193 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:43:59.212404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.212211 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" event={"ID":"091c3f43-8afb-40cb-978c-fffd8b2bd867","Type":"ContainerStarted","Data":"eb276ec1afd22b01787e2ee7dea15ba4c52e0d2ec8bd85eafff6d0f857b7b96e"} Nov 02 23:43:59.213805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.213781 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" event={"ID":"c7815790-5819-412d-a30d-dd0541587c3b","Type":"ContainerStarted","Data":"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592"} Nov 02 23:43:59.213921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.213824 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" event={"ID":"c7815790-5819-412d-a30d-dd0541587c3b","Type":"ContainerStarted","Data":"09bbe2ae73a4eaca55ea46ec76ccd63170efe65135dcdf76a042d546ec789931"} Nov 02 23:43:59.213921 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.213846 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:43:59.331307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:43:59.331255 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" podStartSLOduration=2.331241301 podStartE2EDuration="2.331241301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:59.329726078 +0000 UTC m=+3353.759595942" watchObservedRunningTime="2026-11-02 23:43:59.331241301 +0000 UTC m=+3353.761111162" Nov 02 23:44:00.960468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:00.960411 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" podStartSLOduration=3.960393629 podStartE2EDuration="3.960393629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:59.46267234 +0000 UTC m=+3353.892542207" watchObservedRunningTime="2026-11-02 23:44:00.960393629 +0000 UTC m=+3355.390263540" Nov 02 23:44:00.962345 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:00.962324 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:44:02.589633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:02.589590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:03.609209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:03.608487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:07.557854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:07.557823 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:10.223328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:10.223301 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 02 23:44:10.223809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:10.223611 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 02 23:44:12.453718 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:12.453688 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:14.583592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:14.583555 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:17.431828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:17.431798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:17.562113 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:44:17.562029 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:44:22.339098 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:22.339064 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:27.245137 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:27.245102 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:32.147330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:32.147290 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:34.618636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:34.618596 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:35.263728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.263699 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:44:35.269240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.269220 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.322715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.322683 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-38049c-openshift-default-dockercfg-kn6mr\"" Nov 02 23:44:35.349474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.349438 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.349793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.349740 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.349993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.349973 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.350204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.350187 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.350384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.350370 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.350561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.350545 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.350739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.350723 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.350912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.350895 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lf5b\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.351139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.351125 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.358561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.358521 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:35.378254 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.378225 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:44:35.452994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.452960 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453027 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453067 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453086 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453133 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453202 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453499 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453566 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453568 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453598 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453694 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453637 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lf5b\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.453844 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.453823 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.454237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.454214 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.459439 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.459411 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.464728 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.464703 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.466444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.466425 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.515075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.515002 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.515493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.515468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lf5b\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\") pod \"e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.583281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.583247 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:35.645007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:35.644966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:36.033038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.033013 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:44:36.034052 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:44:36.034025 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721630c6_d15f_4cf6_a3a2_ee47636d0039.slice/crio-1d0ffcf92f40a447bec13933530061bada73aa0e88c5f294525a7d6bddae8ae0 WatchSource:0}: Error finding container 1d0ffcf92f40a447bec13933530061bada73aa0e88c5f294525a7d6bddae8ae0: Status 404 returned error can't find the container with id 1d0ffcf92f40a447bec13933530061bada73aa0e88c5f294525a7d6bddae8ae0 Nov 02 23:44:36.036725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.036686 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:44:36.036853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.036774 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:44:36.036853 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.036823 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:44:36.469793 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.469754 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" event={"ID":"721630c6-d15f-4cf6-a3a2-ee47636d0039","Type":"ContainerStarted","Data":"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c"} Nov 02 23:44:36.469999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.469825 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" event={"ID":"721630c6-d15f-4cf6-a3a2-ee47636d0039","Type":"ContainerStarted","Data":"1d0ffcf92f40a447bec13933530061bada73aa0e88c5f294525a7d6bddae8ae0"} Nov 02 23:44:36.584248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.584219 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:36.593775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.593729 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" podStartSLOduration=1.593716081 podStartE2EDuration="1.593716081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:36.591003783 +0000 UTC m=+3391.020873660" watchObservedRunningTime="2026-11-02 23:44:36.593716081 +0000 UTC m=+3391.023585941" Nov 02 23:44:36.970065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:36.970030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:37.127197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:37.125788 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:37.283351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:37.283267 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:37.641273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:37.634982 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:38.484750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:38.484715 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:38.485960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:38.485938 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:44:42.060147 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:42.060107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:47.038537 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:47.038499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:47.562385 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:44:47.562347 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:44:51.914772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:51.914733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:52.481571 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:52.481542 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:44:56.823628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:44:56.823594 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:00.164726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.164692 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:00.169296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.169277 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.175518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.175491 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.175619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.175532 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwl7b\" (UniqueName: \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.182336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.182315 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:00.186326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.186306 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:00.186442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.186417 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.190761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.190741 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.214999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.214978 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:00.218904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.218888 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:00.219026 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.219009 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.222786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.222771 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.264981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.264948 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:00.276573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276549 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2srf\" (UniqueName: \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.276729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276599 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.276729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276708 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52t66\" (UniqueName: \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.276811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276737 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlc9\" (UniqueName: \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.276850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276827 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbsl\" (UniqueName: \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.276887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276858 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.276922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276883 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.276995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.276979 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.277050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.277037 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.277113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.277093 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwl7b\" (UniqueName: \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.277767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.277749 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.315805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.315770 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 02 23:45:00.320847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.320817 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:00.323701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.323673 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:00.339383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.339354 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwl7b\" (UniqueName: \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.363253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.363209 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:00.366775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.366744 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:00.377970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.377917 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52t66\" (UniqueName: \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.378164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.377973 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlc9\" (UniqueName: \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.378164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378022 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbsl\" (UniqueName: \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.378164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378053 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.378164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378095 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.378164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378152 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.378534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378320 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2srf\" (UniqueName: \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.378534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378398 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.378947 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.378926 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.379031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.379002 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.379166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.379146 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.379490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.379465 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.439968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.439885 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2srf\" (UniqueName: \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\") pod \"maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.440118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.440016 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlc9\" (UniqueName: \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.440307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.440290 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52t66\" (UniqueName: \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\") pod \"maas-api-key-cleanup-29894385-ppm5p\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.440545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.440523 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbsl\" (UniqueName: \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.484159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.484128 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:00.502115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.502091 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:00.512411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.512389 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:00.535327 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.535297 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:00.547532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.543401 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:00.875685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.875663 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:00.876505 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:00.876475 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e15aef1_9f7e_411f_bb72_4ef699ba1093.slice/crio-e1293062797e69063b69e83880124fbbf205cd1777fd6a43d12aa831c6009cf8 WatchSource:0}: Error finding container e1293062797e69063b69e83880124fbbf205cd1777fd6a43d12aa831c6009cf8: Status 404 returned error can't find the container with id e1293062797e69063b69e83880124fbbf205cd1777fd6a43d12aa831c6009cf8 Nov 02 23:45:00.925922 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.925895 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:00.928191 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:00.926377 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d6ecf9_9019_424a_a55e_953b7502a3a9.slice/crio-0382e440389a30372b472a2c6bfe7c425b60fb9a115d1245fe0074c89a93c6c2 WatchSource:0}: Error finding container 0382e440389a30372b472a2c6bfe7c425b60fb9a115d1245fe0074c89a93c6c2: Status 404 returned error can't find the container with id 0382e440389a30372b472a2c6bfe7c425b60fb9a115d1245fe0074c89a93c6c2 Nov 02 23:45:00.929107 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:00.929082 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d79c09c_77bc_4bb8_82ee_a18501cfdb02.slice/crio-4b09df504cc2aa1442f1cab164c1abe309aac9dc344833d18aabea0546c25706 WatchSource:0}: Error finding container 4b09df504cc2aa1442f1cab164c1abe309aac9dc344833d18aabea0546c25706: Status 404 returned error can't find the container with id 4b09df504cc2aa1442f1cab164c1abe309aac9dc344833d18aabea0546c25706 Nov 02 23:45:00.929767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.929744 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:00.977750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:00.977721 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:01.002277 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.002255 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:01.033482 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:01.033457 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e247a7d_4385_4246_99b2_b120d0ddc90f.slice/crio-8f1248cde9bca5c4ee96a8c7264b3e7397cfd7d260d0ca27d1c756148e23326b WatchSource:0}: Error finding container 8f1248cde9bca5c4ee96a8c7264b3e7397cfd7d260d0ca27d1c756148e23326b: Status 404 returned error can't find the container with id 8f1248cde9bca5c4ee96a8c7264b3e7397cfd7d260d0ca27d1c756148e23326b Nov 02 23:45:01.034073 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:01.034051 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e05cdf0_d295_4c85_a427_9ac0ee510e0a.slice/crio-93c33f22145670dff1f6878d98b05f5e840d3d4efc2c9d4abb0c2ffb93da35cc WatchSource:0}: Error finding container 93c33f22145670dff1f6878d98b05f5e840d3d4efc2c9d4abb0c2ffb93da35cc: Status 404 returned error can't find the container with id 93c33f22145670dff1f6878d98b05f5e840d3d4efc2c9d4abb0c2ffb93da35cc Nov 02 23:45:01.427856 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.427819 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 02 23:45:01.432504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.432480 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.486721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.486689 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-882a2a-serving-cert\"" Nov 02 23:45:01.490454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.490427 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 02 23:45:01.490852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.490827 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hjs\" (UniqueName: \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.490962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.490871 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.491004 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.490977 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.491129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.491111 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.592562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.592525 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hjs\" (UniqueName: \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.592749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.592575 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.592749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.592633 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.592836 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.592780 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.592945 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:01.592926 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-serving-cert: secret "maas-api-e2e-shared-a-w4-882a2a-serving-cert" not found Nov 02 23:45:01.593004 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:01.592994 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls podName:16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6 nodeName:}" failed. No retries permitted until 2026-11-02 23:45:02.092974913 +0000 UTC m=+3416.522844759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls") pod "maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" (UID: "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6") : secret "maas-api-e2e-shared-a-w4-882a2a-serving-cert" not found Nov 02 23:45:01.593663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.593632 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.595489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.595459 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.625281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.625241 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:01.654884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.654848 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hjs\" (UniqueName: \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:01.670937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.670897 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerStarted","Data":"6e95557d9949fb24c67e86ccf6b25211e2d0aaf20eb09182eb6f8fc7b83b511b"} Nov 02 23:45:01.671131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.670966 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerStarted","Data":"e1293062797e69063b69e83880124fbbf205cd1777fd6a43d12aa831c6009cf8"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.674127 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerStarted","Data":"a56a547395bf18920e04fd0f06dd64680c7e4a327fa82a1d545623095c7c22b3"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.677315 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerStarted","Data":"0382e440389a30372b472a2c6bfe7c425b60fb9a115d1245fe0074c89a93c6c2"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.677359 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerStarted","Data":"620f591fefeff4eec3094df0d696e7c6e01ce105a2a5a086c67818b8cbc5cd8b"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.677383 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerStarted","Data":"93c33f22145670dff1f6878d98b05f5e840d3d4efc2c9d4abb0c2ffb93da35cc"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.683194 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerStarted","Data":"f0a229c63713e15b24a8a733e6fc70bc250112ea3f305fc44ce067cba6ade82d"} Nov 02 23:45:01.696272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.683274 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerStarted","Data":"4b09df504cc2aa1442f1cab164c1abe309aac9dc344833d18aabea0546c25706"} Nov 02 23:45:01.716769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.716722 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerStarted","Data":"6ecbd9c7991a4943c5599feec1a739d950be24148679fea1613e0414dd0c4686"} Nov 02 23:45:01.716949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.716800 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerStarted","Data":"8f1248cde9bca5c4ee96a8c7264b3e7397cfd7d260d0ca27d1c756148e23326b"} Nov 02 23:45:01.785946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.785897 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:01.789592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.789502 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" podStartSLOduration=1.78948986 podStartE2EDuration="1.78948986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:01.788417733 +0000 UTC m=+3416.218287579" watchObservedRunningTime="2026-11-02 23:45:01.78948986 +0000 UTC m=+3416.219359721" Nov 02 23:45:01.976821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:01.976746 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" podStartSLOduration=1.976724017 podStartE2EDuration="1.976724017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:01.952517938 +0000 UTC m=+3416.382387800" watchObservedRunningTime="2026-11-02 23:45:01.976724017 +0000 UTC m=+3416.406593879" Nov 02 23:45:02.022081 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.022036 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:02.070488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.070431 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" podStartSLOduration=2.070413663 podStartE2EDuration="2.070413663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.068704299 +0000 UTC m=+3416.498574161" watchObservedRunningTime="2026-11-02 23:45:02.070413663 +0000 UTC m=+3416.500283524" Nov 02 23:45:02.111928 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.111894 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:02.114457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.114434 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:02.186206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.186133 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" podStartSLOduration=2.1861166 podStartE2EDuration="2.1861166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.184151012 +0000 UTC m=+3416.614020874" watchObservedRunningTime="2026-11-02 23:45:02.1861166 +0000 UTC m=+3416.615986461" Nov 02 23:45:02.302092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.301985 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" podStartSLOduration=2.301969775 podStartE2EDuration="2.301969775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.299934175 +0000 UTC m=+3416.729804024" watchObservedRunningTime="2026-11-02 23:45:02.301969775 +0000 UTC m=+3416.731839642" Nov 02 23:45:02.348858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.348829 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:02.699330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.699288 4494 generic.go:356] "Generic (PLEG): container finished" podID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerID="a56a547395bf18920e04fd0f06dd64680c7e4a327fa82a1d545623095c7c22b3" exitCode=60 Nov 02 23:45:02.699819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.699371 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerDied","Data":"a56a547395bf18920e04fd0f06dd64680c7e4a327fa82a1d545623095c7c22b3"} Nov 02 23:45:02.699819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.699700 4494 scope.go:122] "RemoveContainer" containerID="a56a547395bf18920e04fd0f06dd64680c7e4a327fa82a1d545623095c7c22b3" Nov 02 23:45:02.701329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.701301 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerID="620f591fefeff4eec3094df0d696e7c6e01ce105a2a5a086c67818b8cbc5cd8b" exitCode=60 Nov 02 23:45:02.701447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.701327 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerDied","Data":"620f591fefeff4eec3094df0d696e7c6e01ce105a2a5a086c67818b8cbc5cd8b"} Nov 02 23:45:02.701605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.701588 4494 scope.go:122] "RemoveContainer" containerID="620f591fefeff4eec3094df0d696e7c6e01ce105a2a5a086c67818b8cbc5cd8b" Nov 02 23:45:02.703082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.703036 4494 generic.go:356] "Generic (PLEG): container finished" podID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerID="f0a229c63713e15b24a8a733e6fc70bc250112ea3f305fc44ce067cba6ade82d" exitCode=60 Nov 02 23:45:02.703146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.703120 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerDied","Data":"f0a229c63713e15b24a8a733e6fc70bc250112ea3f305fc44ce067cba6ade82d"} Nov 02 23:45:02.703347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.703333 4494 scope.go:122] "RemoveContainer" containerID="f0a229c63713e15b24a8a733e6fc70bc250112ea3f305fc44ce067cba6ade82d" Nov 02 23:45:02.705159 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.705139 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerID="6ecbd9c7991a4943c5599feec1a739d950be24148679fea1613e0414dd0c4686" exitCode=60 Nov 02 23:45:02.705287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.705229 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerDied","Data":"6ecbd9c7991a4943c5599feec1a739d950be24148679fea1613e0414dd0c4686"} Nov 02 23:45:02.705457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.705439 4494 scope.go:122] "RemoveContainer" containerID="6ecbd9c7991a4943c5599feec1a739d950be24148679fea1613e0414dd0c4686" Nov 02 23:45:02.706876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.706851 4494 generic.go:356] "Generic (PLEG): container finished" podID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerID="6e95557d9949fb24c67e86ccf6b25211e2d0aaf20eb09182eb6f8fc7b83b511b" exitCode=60 Nov 02 23:45:02.706999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.706920 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerDied","Data":"6e95557d9949fb24c67e86ccf6b25211e2d0aaf20eb09182eb6f8fc7b83b511b"} Nov 02 23:45:02.707160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.707141 4494 scope.go:122] "RemoveContainer" containerID="6e95557d9949fb24c67e86ccf6b25211e2d0aaf20eb09182eb6f8fc7b83b511b" Nov 02 23:45:02.730361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.730332 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 02 23:45:02.732798 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:45:02.732771 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e43dbc_5e5d_44f5_bc84_6dd6f992a1c6.slice/crio-84797508bd602c0b92ecbf3fee6c3b678ce10f83d25a44424f226b8e0183562b WatchSource:0}: Error finding container 84797508bd602c0b92ecbf3fee6c3b678ce10f83d25a44424f226b8e0183562b: Status 404 returned error can't find the container with id 84797508bd602c0b92ecbf3fee6c3b678ce10f83d25a44424f226b8e0183562b Nov 02 23:45:02.734449 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:02.734427 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:45:03.716216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.716158 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerStarted","Data":"4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac"} Nov 02 23:45:03.718285 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.718256 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerStarted","Data":"61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb"} Nov 02 23:45:03.720160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.720135 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" event={"ID":"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6","Type":"ContainerStarted","Data":"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c"} Nov 02 23:45:03.720290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.720219 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:03.720290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.720249 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" event={"ID":"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6","Type":"ContainerStarted","Data":"84797508bd602c0b92ecbf3fee6c3b678ce10f83d25a44424f226b8e0183562b"} Nov 02 23:45:03.722238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.722214 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerStarted","Data":"7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3"} Nov 02 23:45:03.724086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.724066 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerStarted","Data":"54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0"} Nov 02 23:45:03.726804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:03.726782 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerStarted","Data":"9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583"} Nov 02 23:45:04.197495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.197433 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" podStartSLOduration=2.320244921 podStartE2EDuration="3.197418344s" totalImagesPullingTime="877.173423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:04.195063707 +0000 UTC m=+3418.624933567" watchObservedRunningTime="2026-11-02 23:45:04.197418344 +0000 UTC m=+3418.627288205" Nov 02 23:45:04.735008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.734967 4494 generic.go:356] "Generic (PLEG): container finished" podID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerID="61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb" exitCode=60 Nov 02 23:45:04.735685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.735059 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerDied","Data":"61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb"} Nov 02 23:45:04.735685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.735122 4494 scope.go:122] "RemoveContainer" containerID="a56a547395bf18920e04fd0f06dd64680c7e4a327fa82a1d545623095c7c22b3" Nov 02 23:45:04.735685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.735292 4494 scope.go:122] "RemoveContainer" containerID="61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb" Nov 02 23:45:04.735685 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:04.735636 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j_odh-ai-gateway-infra(23d6ecf9-9019-424a-a55e-953b7502a3a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" Nov 02 23:45:04.751876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.751831 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerID="7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3" exitCode=60 Nov 02 23:45:04.752028 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.751960 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerDied","Data":"7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3"} Nov 02 23:45:04.752444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.752421 4494 scope.go:122] "RemoveContainer" containerID="7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3" Nov 02 23:45:04.752768 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:04.752744 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2_odh-ai-gateway-infra(5e05cdf0-d295-4c85-a427-9ac0ee510e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" Nov 02 23:45:04.754870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.754847 4494 generic.go:356] "Generic (PLEG): container finished" podID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerID="54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0" exitCode=60 Nov 02 23:45:04.754959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.754922 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerDied","Data":"54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0"} Nov 02 23:45:04.755154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.755139 4494 scope.go:122] "RemoveContainer" containerID="54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0" Nov 02 23:45:04.755483 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:04.755461 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-ppm5p_odh-ai-gateway-infra(4d79c09c-77bc-4bb8-82ee-a18501cfdb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" Nov 02 23:45:04.756965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.756945 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerID="9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583" exitCode=60 Nov 02 23:45:04.757041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.756967 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerDied","Data":"9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583"} Nov 02 23:45:04.757265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.757251 4494 scope.go:122] "RemoveContainer" containerID="9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583" Nov 02 23:45:04.757523 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:04.757502 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs_odh-ai-gateway-infra(5e247a7d-4385-4246-99b2-b120d0ddc90f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" Nov 02 23:45:04.758970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.758947 4494 generic.go:356] "Generic (PLEG): container finished" podID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerID="4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac" exitCode=60 Nov 02 23:45:04.759044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.758979 4494 scope.go:122] "RemoveContainer" containerID="620f591fefeff4eec3094df0d696e7c6e01ce105a2a5a086c67818b8cbc5cd8b" Nov 02 23:45:04.759044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.759014 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerDied","Data":"4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac"} Nov 02 23:45:04.759246 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.759227 4494 scope.go:122] "RemoveContainer" containerID="4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac" Nov 02 23:45:04.759466 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:04.759447 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68_odh-ai-gateway-infra(4e15aef1-9f7e-411f-bb72-4ef699ba1093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" Nov 02 23:45:04.772077 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.772064 4494 scope.go:122] "RemoveContainer" containerID="f0a229c63713e15b24a8a733e6fc70bc250112ea3f305fc44ce067cba6ade82d" Nov 02 23:45:04.784996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.784980 4494 scope.go:122] "RemoveContainer" containerID="6ecbd9c7991a4943c5599feec1a739d950be24148679fea1613e0414dd0c4686" Nov 02 23:45:04.798189 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:04.798154 4494 scope.go:122] "RemoveContainer" containerID="6e95557d9949fb24c67e86ccf6b25211e2d0aaf20eb09182eb6f8fc7b83b511b" Nov 02 23:45:05.768025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:05.767994 4494 scope.go:122] "RemoveContainer" containerID="61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb" Nov 02 23:45:05.768517 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:05.768364 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j_odh-ai-gateway-infra(23d6ecf9-9019-424a-a55e-953b7502a3a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" Nov 02 23:45:05.769758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:05.769731 4494 scope.go:122] "RemoveContainer" containerID="7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3" Nov 02 23:45:05.770020 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:05.770002 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2_odh-ai-gateway-infra(5e05cdf0-d295-4c85-a427-9ac0ee510e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" Nov 02 23:45:05.771378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:05.771357 4494 scope.go:122] "RemoveContainer" containerID="54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0" Nov 02 23:45:05.771628 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:05.771608 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-ppm5p_odh-ai-gateway-infra(4d79c09c-77bc-4bb8-82ee-a18501cfdb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" Nov 02 23:45:05.773025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:05.773011 4494 scope.go:122] "RemoveContainer" containerID="9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583" Nov 02 23:45:05.773288 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:05.773269 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs_odh-ai-gateway-infra(5e247a7d-4385-4246-99b2-b120d0ddc90f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" Nov 02 23:45:05.774707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:05.774690 4494 scope.go:122] "RemoveContainer" containerID="4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac" Nov 02 23:45:05.774866 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:05.774850 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68_odh-ai-gateway-infra(4e15aef1-9f7e-411f-bb72-4ef699ba1093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" Nov 02 23:45:06.732619 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:06.732583 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:07.277822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:07.277784 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:45:09.140849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:09.140817 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:09.764572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:09.764542 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 02 23:45:11.609831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:11.609790 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:12.084896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:12.084848 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:14.503600 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:14.503561 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:14.646896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:14.646861 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:14.927732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:14.927686 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:16.462687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:16.462648 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:16.687125 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:16.687090 4494 scope.go:122] "RemoveContainer" containerID="7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3" Nov 02 23:45:17.564072 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:17.564032 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:45:17.730602 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.730571 4494 scope.go:122] "RemoveContainer" containerID="9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583" Nov 02 23:45:17.730810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.730748 4494 scope.go:122] "RemoveContainer" containerID="4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac" Nov 02 23:45:17.731542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.731107 4494 scope.go:122] "RemoveContainer" containerID="61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb" Nov 02 23:45:17.731542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.731371 4494 scope.go:122] "RemoveContainer" containerID="54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0" Nov 02 23:45:17.865771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.865739 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerID="71d3a17696062da3ee817f0916d169535e7642548036622940a2564398ddb0dd" exitCode=60 Nov 02 23:45:17.865970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.865807 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerDied","Data":"71d3a17696062da3ee817f0916d169535e7642548036622940a2564398ddb0dd"} Nov 02 23:45:17.865970 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.865866 4494 scope.go:122] "RemoveContainer" containerID="7b4811271a41ff2504bce5f3104584fd0bebf4762668b18df6d73697111d25c3" Nov 02 23:45:17.866280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:17.866248 4494 scope.go:122] "RemoveContainer" containerID="71d3a17696062da3ee817f0916d169535e7642548036622940a2564398ddb0dd" Nov 02 23:45:17.866624 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:17.866605 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2_odh-ai-gateway-infra(5e05cdf0-d295-4c85-a427-9ac0ee510e0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" Nov 02 23:45:18.876797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.876767 4494 generic.go:356] "Generic (PLEG): container finished" podID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerID="eeb49c9c2919e9a216e14ed3436cbe02c44b2eb8ea8efa73b871c1462e4827f6" exitCode=60 Nov 02 23:45:18.877289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.876842 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerDied","Data":"eeb49c9c2919e9a216e14ed3436cbe02c44b2eb8ea8efa73b871c1462e4827f6"} Nov 02 23:45:18.877289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.876935 4494 scope.go:122] "RemoveContainer" containerID="54e6645c5849b0540c71c3ce5291f1c3faf641aef734db08a01921c1bef67ca0" Nov 02 23:45:18.877424 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.877408 4494 scope.go:122] "RemoveContainer" containerID="eeb49c9c2919e9a216e14ed3436cbe02c44b2eb8ea8efa73b871c1462e4827f6" Nov 02 23:45:18.877768 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:18.877738 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-ppm5p_odh-ai-gateway-infra(4d79c09c-77bc-4bb8-82ee-a18501cfdb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" Nov 02 23:45:18.878933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.878912 4494 generic.go:356] "Generic (PLEG): container finished" podID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerID="4ced19da1ed3e85627afe1332d151669656056da687ce25287e1c0d425217992" exitCode=60 Nov 02 23:45:18.879015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.878976 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerDied","Data":"4ced19da1ed3e85627afe1332d151669656056da687ce25287e1c0d425217992"} Nov 02 23:45:18.879411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.879394 4494 scope.go:122] "RemoveContainer" containerID="4ced19da1ed3e85627afe1332d151669656056da687ce25287e1c0d425217992" Nov 02 23:45:18.879710 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:18.879686 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs_odh-ai-gateway-infra(5e247a7d-4385-4246-99b2-b120d0ddc90f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" Nov 02 23:45:18.881232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.881205 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerStarted","Data":"187cd1de93292095c6482346981423a81d7fb573935f37d3d3f9735c40eebbee"} Nov 02 23:45:18.883375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.883343 4494 generic.go:356] "Generic (PLEG): container finished" podID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerID="c089858f878b048981be9686bb53dc107b09ead25791871914948cfb7bf39d62" exitCode=60 Nov 02 23:45:18.883455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.883401 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerDied","Data":"c089858f878b048981be9686bb53dc107b09ead25791871914948cfb7bf39d62"} Nov 02 23:45:18.883737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.883720 4494 scope.go:122] "RemoveContainer" containerID="c089858f878b048981be9686bb53dc107b09ead25791871914948cfb7bf39d62" Nov 02 23:45:18.883975 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:18.883955 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j_odh-ai-gateway-infra(23d6ecf9-9019-424a-a55e-953b7502a3a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" Nov 02 23:45:18.911784 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.911760 4494 scope.go:122] "RemoveContainer" containerID="9c88cdb31db473a0c7f7ef82a2d51386b5d95b97e3f36bd43d74b8f5e3f59583" Nov 02 23:45:18.925453 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.925435 4494 scope.go:122] "RemoveContainer" containerID="61fb546579a577cb889007fba150e31b2807ee64466b614cefbee80e8f6b71eb" Nov 02 23:45:18.992503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:18.992431 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:19.142865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.142834 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:19.183038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.183002 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\") pod \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " Nov 02 23:45:19.183232 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.183104 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\" (UniqueName: \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\") pod \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\" (UID: \"5e05cdf0-d295-4c85-a427-9ac0ee510e0a\") " Nov 02 23:45:19.183365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.183343 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca" pod "5e05cdf0-d295-4c85-a427-9ac0ee510e0a" (UID: "5e05cdf0-d295-4c85-a427-9ac0ee510e0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:19.183720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.183700 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.185478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.185449 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl" pod "5e05cdf0-d295-4c85-a427-9ac0ee510e0a" (UID: "5e05cdf0-d295-4c85-a427-9ac0ee510e0a"). InnerVolumeSpecName "kube-api-access-wnbsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:19.284566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.284488 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbsl\" (UniqueName: \"kubernetes.io/projected/5e05cdf0-d295-4c85-a427-9ac0ee510e0a-kube-api-access-wnbsl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.892672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.892640 4494 generic.go:356] "Generic (PLEG): container finished" podID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerID="187cd1de93292095c6482346981423a81d7fb573935f37d3d3f9735c40eebbee" exitCode=60 Nov 02 23:45:19.893143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.892697 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerDied","Data":"187cd1de93292095c6482346981423a81d7fb573935f37d3d3f9735c40eebbee"} Nov 02 23:45:19.893143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.892757 4494 scope.go:122] "RemoveContainer" containerID="4f730025e3fb0555f29028b674182f5033a704430d8c282b617793e8165b52ac" Nov 02 23:45:19.893143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.893109 4494 scope.go:122] "RemoveContainer" containerID="187cd1de93292095c6482346981423a81d7fb573935f37d3d3f9735c40eebbee" Nov 02 23:45:19.893466 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:19.893441 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68_odh-ai-gateway-infra(4e15aef1-9f7e-411f-bb72-4ef699ba1093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" Nov 02 23:45:19.896257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.896237 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" Nov 02 23:45:19.896354 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.896242 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2" event={"ID":"5e05cdf0-d295-4c85-a427-9ac0ee510e0a","Type":"ContainerDied","Data":"93c33f22145670dff1f6878d98b05f5e840d3d4efc2c9d4abb0c2ffb93da35cc"} Nov 02 23:45:19.905858 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:19.905840 4494 scope.go:122] "RemoveContainer" containerID="71d3a17696062da3ee817f0916d169535e7642548036622940a2564398ddb0dd" Nov 02 23:45:20.005427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.005347 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:20.123250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.123221 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:20.126343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.126318 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:20.132945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.132916 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-980b82-29894385-qrgp2"] Nov 02 23:45:20.163456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.163432 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:20.195385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.195354 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\") pod \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " Nov 02 23:45:20.195553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.195450 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\" (UniqueName: \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\") pod \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\" (UID: \"4d79c09c-77bc-4bb8-82ee-a18501cfdb02\") " Nov 02 23:45:20.195782 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.195749 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca" pod "4d79c09c-77bc-4bb8-82ee-a18501cfdb02" (UID: "4d79c09c-77bc-4bb8-82ee-a18501cfdb02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.196048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.196028 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.197907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.197882 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66" pod "4d79c09c-77bc-4bb8-82ee-a18501cfdb02" (UID: "4d79c09c-77bc-4bb8-82ee-a18501cfdb02"). InnerVolumeSpecName "kube-api-access-52t66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.244590 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.244553 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:20.297341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.297258 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52t66\" (UniqueName: \"kubernetes.io/projected/4d79c09c-77bc-4bb8-82ee-a18501cfdb02-kube-api-access-52t66\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.358451 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.358417 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:20.399616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.399588 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:20.502223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.499812 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\" (UniqueName: \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\") pod \"5e247a7d-4385-4246-99b2-b120d0ddc90f\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " Nov 02 23:45:20.502223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.499866 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\") pod \"5e247a7d-4385-4246-99b2-b120d0ddc90f\" (UID: \"5e247a7d-4385-4246-99b2-b120d0ddc90f\") " Nov 02 23:45:20.502223 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.500479 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca" pod "5e247a7d-4385-4246-99b2-b120d0ddc90f" (UID: "5e247a7d-4385-4246-99b2-b120d0ddc90f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.503005 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.502965 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9" pod "5e247a7d-4385-4246-99b2-b120d0ddc90f" (UID: "5e247a7d-4385-4246-99b2-b120d0ddc90f"). InnerVolumeSpecName "kube-api-access-vqlc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.511145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.511120 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:20.601148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601047 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\" (UniqueName: \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\") pod \"23d6ecf9-9019-424a-a55e-953b7502a3a9\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " Nov 02 23:45:20.601374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601269 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\") pod \"23d6ecf9-9019-424a-a55e-953b7502a3a9\" (UID: \"23d6ecf9-9019-424a-a55e-953b7502a3a9\") " Nov 02 23:45:20.601701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601657 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca" pod "23d6ecf9-9019-424a-a55e-953b7502a3a9" (UID: "23d6ecf9-9019-424a-a55e-953b7502a3a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.602104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601843 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d6ecf9-9019-424a-a55e-953b7502a3a9-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.602104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601863 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqlc9\" (UniqueName: \"kubernetes.io/projected/5e247a7d-4385-4246-99b2-b120d0ddc90f-kube-api-access-vqlc9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.602104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.601877 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e247a7d-4385-4246-99b2-b120d0ddc90f-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.603908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.603882 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf" pod "23d6ecf9-9019-424a-a55e-953b7502a3a9" (UID: "23d6ecf9-9019-424a-a55e-953b7502a3a9"). InnerVolumeSpecName "kube-api-access-f2srf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.702810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.702776 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2srf\" (UniqueName: \"kubernetes.io/projected/23d6ecf9-9019-424a-a55e-953b7502a3a9-kube-api-access-f2srf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.908837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.908752 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" Nov 02 23:45:20.908837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.908784 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j" event={"ID":"23d6ecf9-9019-424a-a55e-953b7502a3a9","Type":"ContainerDied","Data":"0382e440389a30372b472a2c6bfe7c425b60fb9a115d1245fe0074c89a93c6c2"} Nov 02 23:45:20.909406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.908849 4494 scope.go:122] "RemoveContainer" containerID="c089858f878b048981be9686bb53dc107b09ead25791871914948cfb7bf39d62" Nov 02 23:45:20.911819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.911796 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" Nov 02 23:45:20.911988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.911825 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p" event={"ID":"4d79c09c-77bc-4bb8-82ee-a18501cfdb02","Type":"ContainerDied","Data":"4b09df504cc2aa1442f1cab164c1abe309aac9dc344833d18aabea0546c25706"} Nov 02 23:45:20.913408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.913391 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" Nov 02 23:45:20.913530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.913446 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs" event={"ID":"5e247a7d-4385-4246-99b2-b120d0ddc90f","Type":"ContainerDied","Data":"8f1248cde9bca5c4ee96a8c7264b3e7397cfd7d260d0ca27d1c756148e23326b"} Nov 02 23:45:20.922250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.922230 4494 scope.go:122] "RemoveContainer" containerID="eeb49c9c2919e9a216e14ed3436cbe02c44b2eb8ea8efa73b871c1462e4827f6" Nov 02 23:45:20.934553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:20.934517 4494 scope.go:122] "RemoveContainer" containerID="4ced19da1ed3e85627afe1332d151669656056da687ce25287e1c0d425217992" Nov 02 23:45:21.067041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.067017 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:21.108466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.108429 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\") pod \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " Nov 02 23:45:21.108466 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.108467 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\" (UniqueName: \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\") pod \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\" (UID: \"4e15aef1-9f7e-411f-bb72-4ef699ba1093\") " Nov 02 23:45:21.108804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.108782 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca" pod "4e15aef1-9f7e-411f-bb72-4ef699ba1093" (UID: "4e15aef1-9f7e-411f-bb72-4ef699ba1093"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:21.108930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.108917 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15aef1-9f7e-411f-bb72-4ef699ba1093-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.110703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.110678 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b" pod "4e15aef1-9f7e-411f-bb72-4ef699ba1093" (UID: "4e15aef1-9f7e-411f-bb72-4ef699ba1093"). InnerVolumeSpecName "kube-api-access-cwl7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:21.194813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.194701 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:21.198851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.198823 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ec2f9a-29894385-vkmxs"] Nov 02 23:45:21.210470 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.210447 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwl7b\" (UniqueName: \"kubernetes.io/projected/4e15aef1-9f7e-411f-bb72-4ef699ba1093-kube-api-access-cwl7b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.308523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.308478 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:21.313333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.313304 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-875049-29894385-j7j2j"] Nov 02 23:45:21.376960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.376927 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:21.441014 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.440975 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:21.445851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.445785 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-ppm5p"] Nov 02 23:45:21.696989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.696904 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" path="/var/lib/kubelet/pods/23d6ecf9-9019-424a-a55e-953b7502a3a9/volumes" Nov 02 23:45:21.697384 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.697369 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" path="/var/lib/kubelet/pods/4d79c09c-77bc-4bb8-82ee-a18501cfdb02/volumes" Nov 02 23:45:21.697749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.697737 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" path="/var/lib/kubelet/pods/5e05cdf0-d295-4c85-a427-9ac0ee510e0a/volumes" Nov 02 23:45:21.698442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.698426 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" path="/var/lib/kubelet/pods/5e247a7d-4385-4246-99b2-b120d0ddc90f/volumes" Nov 02 23:45:21.924382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.924348 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" Nov 02 23:45:21.924382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.924371 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68" event={"ID":"4e15aef1-9f7e-411f-bb72-4ef699ba1093","Type":"ContainerDied","Data":"e1293062797e69063b69e83880124fbbf205cd1777fd6a43d12aa831c6009cf8"} Nov 02 23:45:21.924935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:21.924451 4494 scope.go:122] "RemoveContainer" containerID="187cd1de93292095c6482346981423a81d7fb573935f37d3d3f9735c40eebbee" Nov 02 23:45:22.093144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:22.093061 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:22.096104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:22.096082 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1f0009-29894385-sgw68"] Nov 02 23:45:23.694426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:23.694391 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" path="/var/lib/kubelet/pods/4e15aef1-9f7e-411f-bb72-4ef699ba1093/volumes" Nov 02 23:45:26.351074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:26.351039 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:31.378100 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:31.378064 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:31.684958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:31.684921 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:32.854199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:32.848788 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:32.992973 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:32.992852 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:36.341628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:36.341582 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:36.718165 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:36.718126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:38.207392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:38.207349 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:45:38.207820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:38.207446 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:45:39.046486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.046454 4494 generic.go:356] "Generic (PLEG): container finished" podID="3640416e-01aa-4410-b91e-8937ad196b86" containerID="2587eb675d601428f2bf03c6e13894d0f4ef2851be554a690cbb88fdcef15d9e" exitCode=0 Nov 02 23:45:39.046734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.046494 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerDied","Data":"2587eb675d601428f2bf03c6e13894d0f4ef2851be554a690cbb88fdcef15d9e"} Nov 02 23:45:39.046937 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.046917 4494 scope.go:122] "RemoveContainer" containerID="2587eb675d601428f2bf03c6e13894d0f4ef2851be554a690cbb88fdcef15d9e" Nov 02 23:45:39.048538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.048509 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:39.048612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.048578 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:39.048612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:39.048609 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:40.063318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:40.063285 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerStarted","Data":"cd974d1c5712cc8e06704ed5533a86838504584622a7770ac837db75f70a583e"} Nov 02 23:45:40.955388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:40.955267 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:45:40.955388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:40.955330 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 02 23:45:40.955388 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:40.955384 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 02 23:45:41.182814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:41.182779 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:41.366767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:41.366681 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:41.961576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:41.961330 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 02 23:45:41.961576 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:41.961386 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 02 23:45:42.660367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.660334 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:45:42.664318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664291 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.664318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664317 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664347 4494 cpu_manager.go:414] "containerMap: removing container" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664355 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664378 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664387 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664401 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664408 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664417 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664424 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664465 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664472 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664486 4494 cpu_manager.go:414] "containerMap: removing container" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664492 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664505 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664512 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664549 4494 cpu_manager.go:414] "containerMap: removing container" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.664550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664556 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664567 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664574 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664584 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664592 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664605 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664613 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664621 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664628 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664785 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664797 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664806 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664815 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664823 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664833 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664841 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664849 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664856 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d79c09c-77bc-4bb8-82ee-a18501cfdb02" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664867 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:45:42.665538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.664879 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:45:42.674518 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.674488 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.716996 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717280 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717441 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717472 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717499 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717692 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717755 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jpgj\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.717821 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.717792 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.728711 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.728688 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3058ee-openshift-default-dockercfg-gk2j9\"" Nov 02 23:45:42.756798 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.756761 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:42.784967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.784931 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:45:42.818692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818655 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.818885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818701 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.818885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818728 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpgj\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.818885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818759 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.818885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818857 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.818899 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819035 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819063 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819088 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819149 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819258 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819303 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819394 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.819685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.819665 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.821405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.821384 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.821675 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.821656 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.902185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.902130 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.904099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.903989 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpgj\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\") pod \"e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:42.904346 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.904141 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:42.952806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.952701 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 02 23:45:42.953141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.953107 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 02 23:45:42.991407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:42.991377 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:43.413473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.413442 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:43.461103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.461056 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:45:43.496502 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.496459 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:43.497093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.497065 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:43.497300 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.497279 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:45:43.956196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.955635 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 02 23:45:43.956196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:43.955717 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 02 23:45:44.099199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.099123 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerStarted","Data":"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661"} Nov 02 23:45:44.099372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.099224 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerStarted","Data":"2111f0a9ced83b5aaada64e477de64a902ba297c0329def4e05d9935bfb708a8"} Nov 02 23:45:44.223617 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.223523 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podStartSLOduration=2.223506589 podStartE2EDuration="2.223506589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:44.221487645 +0000 UTC m=+3458.651357542" watchObservedRunningTime="2026-11-02 23:45:44.223506589 +0000 UTC m=+3458.653376451" Nov 02 23:45:44.955639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.955598 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 02 23:45:44.955866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.955666 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 02 23:45:44.991889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.991854 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:44.998063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:44.998040 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:45.106093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:45.106061 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:45.106955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:45.106934 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:45:45.954827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:45.954798 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:45:45.955109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:45.955089 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:45:45.955850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:45.955833 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 02 23:45:46.276560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:46.276468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:47.128796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:47.128764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:47.295366 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:47.295333 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:47.447461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:47.447429 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:47.561504 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:47.561425 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:45:48.312146 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:48.312111 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:49.796266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:49.796232 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:45:50.325812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:50.325777 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:50.331326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:50.331305 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:45:50.331534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:50.331513 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerName="istio-proxy" containerID="cri-o://09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c" gracePeriod=30 Nov 02 23:45:50.667673 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:50.667636 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:50.811374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:50.811339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:51.539933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:51.539899 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:53.487484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:53.487452 4494 patch_prober.go:29] interesting pod/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:45:53.487873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:53.487514 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:45:53.630692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:53.630661 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:53.801383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:53.801302 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:55.648387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.648359 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:45:55.721554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721517 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.721554 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721553 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.721857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721585 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.721857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721718 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.721857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721759 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.721857 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721782 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.722144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721874 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.722144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.721903 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.722144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722028 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:55.722144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722101 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\") pod \"721630c6-d15f-4cf6-a3a2-ee47636d0039\" (UID: \"721630c6-d15f-4cf6-a3a2-ee47636d0039\") " Nov 02 23:45:55.722447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722327 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:55.722628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722606 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:55.722702 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722664 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:55.722751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722726 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721630c6-d15f-4cf6-a3a2-ee47636d0039-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.722835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722747 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.722835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722762 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.722959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.722931 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:55.724130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.724099 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:55.724749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.724720 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:55.724847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.724775 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:45:55.724898 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.724887 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b" pod "721630c6-d15f-4cf6-a3a2-ee47636d0039" (UID: "721630c6-d15f-4cf6-a3a2-ee47636d0039"). InnerVolumeSpecName "kube-api-access-4lf5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:55.824443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824412 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.824443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824443 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.824682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824457 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.824682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824470 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721630c6-d15f-4cf6-a3a2-ee47636d0039-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.824682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824482 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721630c6-d15f-4cf6-a3a2-ee47636d0039-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.824682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.824494 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lf5b\" (UniqueName: \"kubernetes.io/projected/721630c6-d15f-4cf6-a3a2-ee47636d0039-kube-api-access-4lf5b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:45:55.893683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:55.893597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:56.208908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.208870 4494 generic.go:356] "Generic (PLEG): container finished" podID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerID="09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c" exitCode=0 Nov 02 23:45:56.209114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.208914 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" Nov 02 23:45:56.209114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.208957 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" event={"ID":"721630c6-d15f-4cf6-a3a2-ee47636d0039","Type":"ContainerDied","Data":"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c"} Nov 02 23:45:56.209114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.209016 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk" event={"ID":"721630c6-d15f-4cf6-a3a2-ee47636d0039","Type":"ContainerDied","Data":"1d0ffcf92f40a447bec13933530061bada73aa0e88c5f294525a7d6bddae8ae0"} Nov 02 23:45:56.209114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.209044 4494 scope.go:122] "RemoveContainer" containerID="09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c" Nov 02 23:45:56.220323 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.220298 4494 scope.go:122] "RemoveContainer" containerID="09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c" Nov 02 23:45:56.220592 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:45:56.220572 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c\": container with ID starting with 09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c not found: ID does not exist" containerID="09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c" Nov 02 23:45:56.220661 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.220603 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c"} err="failed to get container status \"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c\": rpc error: code = NotFound desc = could not find container \"09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c\": container with ID starting with 09d4de49675aa72f72143c242e61c80030646a5ee005e1d4e482dc39eca19a2c not found: ID does not exist" Nov 02 23:45:56.379721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.379691 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:45:56.384021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.383997 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-38049c-openshift-default-57f5d98cc9-qdmzk"] Nov 02 23:45:56.435674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:56.435639 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:57.694691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:57.694656 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" path="/var/lib/kubelet/pods/721630c6-d15f-4cf6-a3a2-ee47636d0039/volumes" Nov 02 23:45:58.909299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:58.909259 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:45:59.813623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:45:59.813569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:01.353145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:01.353115 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:05.933861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:05.933829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:06.249139 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:06.249057 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:06.578966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:06.578607 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:06.861556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:06.861461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:10.183490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:10.183453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:10.361681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:10.361651 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:10.509338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:10.509243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:11.427648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:11.427614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:13.381037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:13.381004 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:13.559724 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:13.559693 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:14.191024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:14.190990 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:14.335883 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:14.335851 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:14.607503 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:14.607147 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:46:14.859588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:14.859504 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:15.011802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:15.011761 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:15.276347 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:15.276313 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:16.746979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:16.746924 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:17.561244 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:46:17.561210 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:46:19.610390 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:19.610356 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" podUID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" containerName="payload-pre-processing" containerID="cri-o://779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5" gracePeriod=25 Nov 02 23:46:19.868741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:19.868668 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:46:20.002049 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.002016 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\" (UniqueName: \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\") pod \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " Nov 02 23:46:20.002284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.002073 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\") pod \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\" (UID: \"7a56e2ad-6d75-4520-aab0-fa77d6c9744b\") " Nov 02 23:46:20.002561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.002534 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume" pod "7a56e2ad-6d75-4520-aab0-fa77d6c9744b" (UID: "7a56e2ad-6d75-4520-aab0-fa77d6c9744b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:46:20.002683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.002667 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:20.004404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.004379 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b" pod "7a56e2ad-6d75-4520-aab0-fa77d6c9744b" (UID: "7a56e2ad-6d75-4520-aab0-fa77d6c9744b"). InnerVolumeSpecName "kube-api-access-rgx8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:20.010521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.010491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:20.103902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.103859 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgx8b\" (UniqueName: \"kubernetes.io/projected/7a56e2ad-6d75-4520-aab0-fa77d6c9744b-kube-api-access-rgx8b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:46:20.371969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.371935 4494 generic.go:356] "Generic (PLEG): container finished" podID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" containerID="779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5" exitCode=0 Nov 02 23:46:20.372140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.371998 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" Nov 02 23:46:20.372140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.372033 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" event={"ID":"7a56e2ad-6d75-4520-aab0-fa77d6c9744b","Type":"ContainerDied","Data":"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5"} Nov 02 23:46:20.372140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.372091 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2" event={"ID":"7a56e2ad-6d75-4520-aab0-fa77d6c9744b","Type":"ContainerDied","Data":"9445ba28c096511480f6f8dfc37acd8910d1df159751de921aa46ebf84331d94"} Nov 02 23:46:20.372140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.372118 4494 scope.go:122] "RemoveContainer" containerID="779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5" Nov 02 23:46:20.382019 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.382001 4494 scope.go:122] "RemoveContainer" containerID="779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5" Nov 02 23:46:20.382300 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:46:20.382282 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5\": container with ID starting with 779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5 not found: ID does not exist" containerID="779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5" Nov 02 23:46:20.382365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.382306 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5"} err="failed to get container status \"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5\": rpc error: code = NotFound desc = could not find container \"779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5\": container with ID starting with 779e454a0460eb42b17ed15bfa01ca76b3f765f8686cc7a3c1cb9054a74b73a5 not found: ID does not exist" Nov 02 23:46:20.550200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.550133 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:46:20.590240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:20.590208 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-875049-6c57c68c5-gskf2"] Nov 02 23:46:21.693904 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:21.693866 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" path="/var/lib/kubelet/pods/7a56e2ad-6d75-4520-aab0-fa77d6c9744b/volumes" Nov 02 23:46:21.819588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:21.819555 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:22.310230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:22.310192 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:22.745682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:22.745634 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:23.388433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:23.388396 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:23.702447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:23.702418 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:26.877713 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:26.877678 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:27.305690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:27.305657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:27.482314 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:27.482276 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:27.628748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:27.628666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:30.551114 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:30.551082 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:30.731157 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:30.729489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:32.077642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:32.077607 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:33.192840 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:33.192804 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:34.711208 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:34.711142 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:35.212381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.212343 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:46:35.213051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213010 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:46:35.213051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213045 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213079 4494 cpu_manager.go:414] "containerMap: removing container" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerName="istio-proxy" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213085 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerName="istio-proxy" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213106 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213115 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213143 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" containerName="payload-pre-processing" Nov 02 23:46:35.213191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213151 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" containerName="payload-pre-processing" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213305 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e05cdf0-d295-4c85-a427-9ac0ee510e0a" containerName="cleanup" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213324 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="721630c6-d15f-4cf6-a3a2-ee47636d0039" containerName="istio-proxy" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213333 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15aef1-9f7e-411f-bb72-4ef699ba1093" containerName="cleanup" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213340 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d6ecf9-9019-424a-a55e-953b7502a3a9" containerName="cleanup" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213346 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e247a7d-4385-4246-99b2-b120d0ddc90f" containerName="cleanup" Nov 02 23:46:35.213404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.213351 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a56e2ad-6d75-4520-aab0-fa77d6c9744b" containerName="payload-pre-processing" Nov 02 23:46:35.218474 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.218453 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.278634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.278601 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f49ab-openshift-default-dockercfg-67bb9\"" Nov 02 23:46:35.280683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280647 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.280850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280775 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.280850 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280824 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.280979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280866 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.280979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280927 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.281101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.280982 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.281101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.281042 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.281101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.281075 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.281299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.281122 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhrt\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.316758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.316719 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:35.343525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.343490 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.384638 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.384798 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385007 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385050 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385118 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385195 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385256 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385289 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.395511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.385313 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhrt\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.411552 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.397913 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.411812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.408873 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.412674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.409133 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.440623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.409398 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.440824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.410003 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.440915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.414003 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.443426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.443399 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.447257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.447225 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhrt\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.459710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.459686 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\") pod \"e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.532471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.532388 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:35.636472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.633567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:35.972356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.972323 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:46:35.972758 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:46:35.972551 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc39ed6_e3c7_46c6_a2fe_fdba708bc268.slice/crio-95c09ad0596d4924fe9f37bc128d0c2936e0d9113d933a8dcebacef66215ce31 WatchSource:0}: Error finding container 95c09ad0596d4924fe9f37bc128d0c2936e0d9113d933a8dcebacef66215ce31: Status 404 returned error can't find the container with id 95c09ad0596d4924fe9f37bc128d0c2936e0d9113d933a8dcebacef66215ce31 Nov 02 23:46:35.975074 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.975041 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:35.975230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.975117 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:35.975230 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:35.975165 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:46:36.496976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:36.496932 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerStarted","Data":"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0"} Nov 02 23:46:36.497216 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:36.497023 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerStarted","Data":"95c09ad0596d4924fe9f37bc128d0c2936e0d9113d933a8dcebacef66215ce31"} Nov 02 23:46:36.532790 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:36.532755 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:36.619481 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:36.619430 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podStartSLOduration=1.619415492 podStartE2EDuration="1.619415492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:46:36.61747053 +0000 UTC m=+3511.047340382" watchObservedRunningTime="2026-11-02 23:46:36.619415492 +0000 UTC m=+3511.049285352" Nov 02 23:46:37.514541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:37.514491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:37.539360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:37.539326 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:38.509393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:38.509365 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:38.510313 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:38.510293 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:46:39.465284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:39.465239 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:39.625514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:39.625480 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:39.780893 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:39.780794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:40.116267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:40.116151 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:46:42.634861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:42.634825 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:42.781116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:42.781078 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:42.983529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:42.983494 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:47.560851 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:46:47.560813 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:46:48.098881 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:48.098843 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:51.018150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:51.018112 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:51.163912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:51.163882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:53.388806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:53.388773 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:54.371308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:54.371272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:54.554333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:54.554302 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:58.317831 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:58.317791 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:58.474752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:58.474692 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:46:58.620380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:46:58.620301 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:03.605351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:03.605308 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:08.806479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:08.806440 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:13.944915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:13.944862 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:14.729698 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:14.729668 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:47:15.638761 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:15.638732 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:17.590090 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:47:17.590006 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:47:18.863813 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:18.863757 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:23.698966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:23.698913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:23.877162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:23.877126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:24.055597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:24.055515 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:25.819122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.819090 4494 generic.go:356] "Generic (PLEG): container finished" podID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerID="7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661" exitCode=0 Nov 02 23:47:25.819588 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.819146 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerDied","Data":"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661"} Nov 02 23:47:25.819633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.819590 4494 scope.go:122] "RemoveContainer" containerID="7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661" Nov 02 23:47:25.821308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.821282 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:25.821382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.821365 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:25.821422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:25.821398 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:47:26.826919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:26.826886 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerStarted","Data":"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae"} Nov 02 23:47:26.992250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:26.992214 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:47:26.993293 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:26.993269 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 23:47:26.993449 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:26.993320 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 23:47:27.992639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:27.992594 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 23:47:27.993150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:27.992658 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 23:47:28.808671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:28.808622 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:28.991837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:28.991800 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 23:47:28.992035 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:28.991849 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 23:47:29.992234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:29.992199 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 23:47:29.992655 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:29.992252 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 23:47:30.106043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:30.106005 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:47:30.992696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:30.992662 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Nov 02 23:47:30.993163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:30.992725 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Nov 02 23:47:32.072217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:32.072161 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:47:32.074288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:32.074264 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:47:33.719342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:33.719311 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:38.563304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:38.563271 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:43.425299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:43.425264 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:43.570820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:43.570786 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:43.714656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:43.714577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:44.836720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:44.836687 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:47:47.564266 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:47:47.564227 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:47:48.458342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:48.458310 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:53.435865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:53.435829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:47:58.353064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:47:58.353028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:02.901226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.901192 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:48:02.905416 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.905399 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:02.969090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.969059 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:02.969306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.969131 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9zl\" (UniqueName: \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:02.969306 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.969261 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:02.969536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:02.969514 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.010441 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.010413 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0f49ab-serving-cert\"" Nov 02 23:48:03.013689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.013668 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:48:03.070994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.070964 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.071186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.071042 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.071186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.071081 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9zl\" (UniqueName: \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.071186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.071106 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.071373 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:48:03.071267 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-serving-cert: secret "maas-api-e2e-extmod-a-0f49ab-serving-cert" not found Nov 02 23:48:03.071373 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:48:03.071329 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls podName:cce4c8b1-ceb4-46c4-ab52-dc547651ed66 nodeName:}" failed. No retries permitted until 2026-11-02 23:48:03.571310974 +0000 UTC m=+3598.001180831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls") pod "maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" (UID: "cce4c8b1-ceb4-46c4-ab52-dc547651ed66") : secret "maas-api-e2e-extmod-a-0f49ab-serving-cert" not found Nov 02 23:48:03.071788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.071765 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.073948 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.073915 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.100545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.100519 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:03.133418 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.133395 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9zl\" (UniqueName: \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.272209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.269913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:03.414539 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.414497 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:03.579971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.579877 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.582501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.582475 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:03.818894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:03.818859 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:04.205584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:04.205553 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:48:04.207020 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:48:04.206977 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce4c8b1_ceb4_46c4_ab52_dc547651ed66.slice/crio-982f6fee18ec5311628b2cfd1cf6deb784d89586637ffb549135fef29dc50c00 WatchSource:0}: Error finding container 982f6fee18ec5311628b2cfd1cf6deb784d89586637ffb549135fef29dc50c00: Status 404 returned error can't find the container with id 982f6fee18ec5311628b2cfd1cf6deb784d89586637ffb549135fef29dc50c00 Nov 02 23:48:05.158580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:05.158548 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" event={"ID":"cce4c8b1-ceb4-46c4-ab52-dc547651ed66","Type":"ContainerStarted","Data":"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0"} Nov 02 23:48:05.158767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:05.158605 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" event={"ID":"cce4c8b1-ceb4-46c4-ab52-dc547651ed66","Type":"ContainerStarted","Data":"982f6fee18ec5311628b2cfd1cf6deb784d89586637ffb549135fef29dc50c00"} Nov 02 23:48:05.158767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:05.158681 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:05.307545 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:05.307492 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" podStartSLOduration=2.438187113 podStartE2EDuration="3.307477154s" totalImagesPullingTime="869.290041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:05.306416228 +0000 UTC m=+3599.736286099" watchObservedRunningTime="2026-11-02 23:48:05.307477154 +0000 UTC m=+3599.737347043" Nov 02 23:48:08.244909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:08.244870 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:11.170786 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:11.170754 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:48:13.100239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:13.100201 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:15.984489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:15.984457 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:17.560194 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:48:17.560146 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:48:17.970785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:17.970740 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:22.587340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:22.587304 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:48:22.939509 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:22.939456 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:28.039905 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:28.039807 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:29.903374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:29.903295 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:30.572465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:30.572430 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:31.817003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:31.816971 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:32.466667 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.466595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:32.565298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.561676 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:48:32.565298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.561909 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" podUID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" containerName="maas-api" containerID="cri-o://40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493" gracePeriod=30 Nov 02 23:48:32.668979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.668940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:32.832092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.832053 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:32.882865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.882810 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:48:32.977087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.974603 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\") pod \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " Nov 02 23:48:32.977087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.974690 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\" (UniqueName: \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\") pod \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " Nov 02 23:48:32.977087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.974892 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\") pod \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " Nov 02 23:48:32.977087 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.974946 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\") pod \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\" (UID: \"680e76ef-c2ea-48e2-8bf0-4b176eea4176\") " Nov 02 23:48:32.977729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.977685 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca" pod "680e76ef-c2ea-48e2-8bf0-4b176eea4176" (UID: "680e76ef-c2ea-48e2-8bf0-4b176eea4176"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:48:32.986616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.986567 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls" pod "680e76ef-c2ea-48e2-8bf0-4b176eea4176" (UID: "680e76ef-c2ea-48e2-8bf0-4b176eea4176"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:48:32.990015 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.989946 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs" pod "680e76ef-c2ea-48e2-8bf0-4b176eea4176" (UID: "680e76ef-c2ea-48e2-8bf0-4b176eea4176"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:48:32.992350 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:32.992318 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j" pod "680e76ef-c2ea-48e2-8bf0-4b176eea4176" (UID: "680e76ef-c2ea-48e2-8bf0-4b176eea4176"). InnerVolumeSpecName "kube-api-access-tsv7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:48:33.077233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.077202 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:33.077233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.077231 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/680e76ef-c2ea-48e2-8bf0-4b176eea4176-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:33.077458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.077244 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/680e76ef-c2ea-48e2-8bf0-4b176eea4176-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:33.077458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.077253 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsv7j\" (UniqueName: \"kubernetes.io/projected/680e76ef-c2ea-48e2-8bf0-4b176eea4176-kube-api-access-tsv7j\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:48:33.492565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.492536 4494 generic.go:356] "Generic (PLEG): container finished" podID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerID="696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0" exitCode=0 Nov 02 23:48:33.492804 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.492609 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerDied","Data":"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0"} Nov 02 23:48:33.493076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.493057 4494 scope.go:122] "RemoveContainer" containerID="696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0" Nov 02 23:48:33.494153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494132 4494 generic.go:356] "Generic (PLEG): container finished" podID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" containerID="40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493" exitCode=0 Nov 02 23:48:33.494289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494205 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" Nov 02 23:48:33.494289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494216 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" event={"ID":"680e76ef-c2ea-48e2-8bf0-4b176eea4176","Type":"ContainerDied","Data":"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493"} Nov 02 23:48:33.494289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494280 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9" event={"ID":"680e76ef-c2ea-48e2-8bf0-4b176eea4176","Type":"ContainerDied","Data":"0253cc879c869a324dcd09dd68c631bd01ca204ca6189c1cc31b3e145baa66e0"} Nov 02 23:48:33.494468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494316 4494 scope.go:122] "RemoveContainer" containerID="40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493" Nov 02 23:48:33.494931 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494902 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.495020 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.494995 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.495070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.495050 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:33.509662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.509620 4494 scope.go:122] "RemoveContainer" containerID="40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493" Nov 02 23:48:33.509988 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:48:33.509960 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493\": container with ID starting with 40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493 not found: ID does not exist" containerID="40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493" Nov 02 23:48:33.510075 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.510000 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493"} err="failed to get container status \"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493\": rpc error: code = NotFound desc = could not find container \"40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493\": container with ID starting with 40a7f245cb029732660bc62ffe860fcefd0bcddb31353a405ec5c9ff6628f493 not found: ID does not exist" Nov 02 23:48:33.746251 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.746160 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:48:33.749685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:33.749649 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-875049-6675dc6685-wwpd9"] Nov 02 23:48:34.404298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.404260 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:34.502614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.502581 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerStarted","Data":"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e"} Nov 02 23:48:34.536197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.535618 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:48:34.543195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.536903 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:34.543195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.536966 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:34.559194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:34.552618 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:35.056859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:35.056815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:35.558290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:35.557771 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:35.558290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:35.557833 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:35.667701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:35.667666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:35.694593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:35.694549 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" path="/var/lib/kubelet/pods/680e76ef-c2ea-48e2-8bf0-4b176eea4176/volumes" Nov 02 23:48:36.259896 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.259858 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:48:36.260627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.260608 4494 cpu_manager.go:414] "containerMap: removing container" podUID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" containerName="maas-api" Nov 02 23:48:36.260627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.260627 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" containerName="maas-api" Nov 02 23:48:36.260849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.260834 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e76ef-c2ea-48e2-8bf0-4b176eea4176" containerName="maas-api" Nov 02 23:48:36.265532 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.265513 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.495971 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496019 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496065 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqn9s\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496223 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496292 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496331 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496408 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496477 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.496507 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.499204 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.499059 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-d75a69-openshift-default-dockercfg-9n8f8\"" Nov 02 23:48:36.502003 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.501833 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:48:36.503758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.503727 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:36.555116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.555021 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:36.555443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.555413 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:36.597294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597262 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597331 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597362 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597429 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597492 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597521 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597695 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.597800 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597755 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597807 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqn9s\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597914 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.597964 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.598138 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.598191 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.598447 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.598343 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.599957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.599938 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.600071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.599989 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.659380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.659325 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.660197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.660153 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqn9s\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\") pod \"e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:36.663636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.663591 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:36.811259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.811033 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:36.880955 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:36.880924 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:37.357629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.352745 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:48:37.360433 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:48:37.360370 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bebd28_eaef_4ceb_84a3_b22b763b295c.slice/crio-0d769a815699c2a2f48469886e993cd67d6c6ec0f630db3d84e52db4727f14ea WatchSource:0}: Error finding container 0d769a815699c2a2f48469886e993cd67d6c6ec0f630db3d84e52db4727f14ea: Status 404 returned error can't find the container with id 0d769a815699c2a2f48469886e993cd67d6c6ec0f630db3d84e52db4727f14ea Nov 02 23:48:37.363510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.363477 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:37.363592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.363546 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:37.363592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.363577 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:48:37.528101 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.528058 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerStarted","Data":"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e"} Nov 02 23:48:37.528317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.528117 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerStarted","Data":"0d769a815699c2a2f48469886e993cd67d6c6ec0f630db3d84e52db4727f14ea"} Nov 02 23:48:37.560080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.538538 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:37.560080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.538593 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:37.658626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.658500 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podStartSLOduration=1.6584807499999998 podStartE2EDuration="1.65848075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:37.657804679 +0000 UTC m=+3632.087674545" watchObservedRunningTime="2026-11-02 23:48:37.65848075 +0000 UTC m=+3632.088350615" Nov 02 23:48:37.882051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:37.882019 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:38.510056 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.510021 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:48:38.552205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.540529 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:38.552205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.540602 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:38.612148 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.612111 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:38.758380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.758340 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:38.888598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:38.888522 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:39.539906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.539873 4494 patch_prober.go:29] interesting pod/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Nov 02 23:48:39.546649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.546231 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Nov 02 23:48:39.571639 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.571182 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:39.572072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.572052 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:48:39.636421 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.636362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:39.809923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.809835 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:39.959610 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:39.959569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:40.537824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:40.537793 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:48:40.538872 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:40.538846 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:48:41.937121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:41.937088 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:42.133598 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:42.133553 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:42.277461 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:42.277382 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:45.193909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:45.193864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:45.370213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:45.370144 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:45.516685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:45.516580 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:47.559884 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:48:47.559798 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:48:50.714671 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:50.714634 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:56.168462 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:56.168424 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:48:58.766605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:48:58.766576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:00.053520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.053468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:00.562278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.562246 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:49:00.567108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.567087 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.620432 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.620394 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-056584-openshift-default-dockercfg-lrgc5\"" Nov 02 23:49:00.654894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.654851 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:00.675926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.675895 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:49:00.749195 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.723013 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.749546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.749514 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.749749 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.749731 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.753305 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.753272 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.792257 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.792211 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwfk\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.792580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.792558 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.792788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.792769 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.792924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.792901 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.793772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.793745 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.858494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.858414 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:00.895280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895242 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895280 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895284 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895347 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895389 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895427 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwfk\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895474 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895550 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895573 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895886 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895729 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.895985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895948 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.896060 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.895978 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.896130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.896098 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.896250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.896228 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.898287 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.898266 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.898411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.898391 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.957443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.957407 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:00.957638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:00.957612 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwfk\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\") pod \"e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:01.181995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.181960 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:01.534868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.534750 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:01.618334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.618294 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:49:01.621801 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:49:01.621774 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd664b1cd_f0a0_43a4_b294_ab1f7cde0ef6.slice/crio-7997c47b1b63e92ffe9fa44a272521d40693452c7c69aab70eafd483343b9f76 WatchSource:0}: Error finding container 7997c47b1b63e92ffe9fa44a272521d40693452c7c69aab70eafd483343b9f76: Status 404 returned error can't find the container with id 7997c47b1b63e92ffe9fa44a272521d40693452c7c69aab70eafd483343b9f76 Nov 02 23:49:01.624567 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.624526 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:49:01.624649 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.624604 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:49:01.624689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.624651 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:49:01.719990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.719948 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerStarted","Data":"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5"} Nov 02 23:49:01.720166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.720008 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerStarted","Data":"7997c47b1b63e92ffe9fa44a272521d40693452c7c69aab70eafd483343b9f76"} Nov 02 23:49:01.845912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:01.845789 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podStartSLOduration=1.8457745779999999 podStartE2EDuration="1.845774578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:01.841712094 +0000 UTC m=+3656.271581981" watchObservedRunningTime="2026-11-02 23:49:01.845774578 +0000 UTC m=+3656.275644438" Nov 02 23:49:02.182367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:02.182331 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:02.187780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:02.187758 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:02.726891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:02.726859 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:02.727987 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:02.727961 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:49:04.558318 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:04.558286 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:04.720848 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:04.720809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:04.867579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:04.867478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:06.779301 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:06.779255 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:07.794332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:07.794301 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:07.974023 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:07.973950 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:11.821027 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:11.820989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:14.799517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:14.799488 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:49:16.330297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:16.330257 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:16.738339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:16.738303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:17.559882 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:49:17.559840 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:49:21.191659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:21.191620 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:22.091534 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:22.091501 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:24.950878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:24.950844 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:27.238870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:27.238830 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:32.103516 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:32.103460 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:37.135222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:37.135162 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:38.406392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:38.406349 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:42.443785 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:42.443753 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:44.669380 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:44.669336 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:45.307261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:45.307229 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:45.451167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:45.451137 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:47.561320 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:49:47.561277 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:49:47.641788 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:47.641755 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:48.904656 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:48.904621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:50.407090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:50.407049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:52.853315 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:52.853277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:49:58.131962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:49:58.131928 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:02.060206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:02.060153 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:50:03.401308 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:03.401273 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:03.559923 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:03.559885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:03.707651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:03.707618 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:08.539399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:08.539362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:13.378792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:13.378743 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:13.534866 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:13.534826 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:13.698256 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:13.698225 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:16.678915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:16.678881 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:17.561575 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:50:17.561492 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:50:18.439524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:18.439489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:23.378710 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:23.378671 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:28.249629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:28.249591 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:33.143274 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:33.143238 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:38.035141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:38.035098 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:39.146864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:39.146827 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:42.600264 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.600235 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:50:42.604365 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.604343 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.658456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.658415 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-056584-serving-cert\"" Nov 02 23:50:42.713017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.712989 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:50:42.760054 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.760020 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.760250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.760212 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mkf\" (UniqueName: \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.760361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.760333 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.760497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.760479 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.791249 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.791224 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:42.862771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.862678 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.862960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.862832 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mkf\" (UniqueName: \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.862960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.862916 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.863086 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.863055 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.865242 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:50:42.863279 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-serving-cert: secret "maas-api-e2e-extmod-b-056584-serving-cert" not found Nov 02 23:50:42.865242 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:50:42.863354 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls podName:9db42416-5e58-4da3-b6c5-d8393d0a97e9 nodeName:}" failed. No retries permitted until 2026-11-02 23:50:43.363329683 +0000 UTC m=+3757.793199526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls") pod "maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" (UID: "9db42416-5e58-4da3-b6c5-d8393d0a97e9") : secret "maas-api-e2e-extmod-b-056584-serving-cert" not found Nov 02 23:50:42.865242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.864345 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.871356 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.870283 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.925498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.925464 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mkf\" (UniqueName: \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:42.949231 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:42.949191 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:43.094740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.094705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:43.370370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.370335 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:43.372968 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.372950 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-056584-75cc65df45-qdksq\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:43.518105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.518076 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:43.903613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.903581 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:50:43.904068 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:50:43.904043 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db42416_5e58_4da3_b6c5_d8393d0a97e9.slice/crio-d51bc56108d712005295f4b73770f6a1859bb4bb69fb239522978b3300fb8cfd WatchSource:0}: Error finding container d51bc56108d712005295f4b73770f6a1859bb4bb69fb239522978b3300fb8cfd: Status 404 returned error can't find the container with id d51bc56108d712005295f4b73770f6a1859bb4bb69fb239522978b3300fb8cfd Nov 02 23:50:43.905492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:43.905473 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:50:44.463936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:44.463900 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" event={"ID":"9db42416-5e58-4da3-b6c5-d8393d0a97e9","Type":"ContainerStarted","Data":"d51bc56108d712005295f4b73770f6a1859bb4bb69fb239522978b3300fb8cfd"} Nov 02 23:50:45.472055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:45.472015 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" event={"ID":"9db42416-5e58-4da3-b6c5-d8393d0a97e9","Type":"ContainerStarted","Data":"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27"} Nov 02 23:50:45.472584 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:45.472182 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:45.607550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:45.607488 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" podStartSLOduration=2.729561008 podStartE2EDuration="3.60746091s" totalImagesPullingTime="877.899902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:45.590657263 +0000 UTC m=+3760.020527133" watchObservedRunningTime="2026-11-02 23:50:45.60746091 +0000 UTC m=+3760.037330788" Nov 02 23:50:47.562145 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:50:47.562103 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:50:47.817045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:47.816913 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:51.483538 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:51.483506 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:50:52.664357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:52.664313 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:50:53.210805 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:53.210762 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:50:57.653201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:50:57.649585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:02.660083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:02.660045 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:03.620707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.620609 4494 generic.go:356] "Generic (PLEG): container finished" podID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerID="9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e" exitCode=0 Nov 02 23:51:03.620707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.620682 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerDied","Data":"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e"} Nov 02 23:51:03.621092 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.621078 4494 scope.go:122] "RemoveContainer" containerID="9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e" Nov 02 23:51:03.622816 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.622789 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:03.622894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.622862 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:03.622942 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:03.622896 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:04.629735 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:04.629702 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerStarted","Data":"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e"} Nov 02 23:51:04.881605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:04.881515 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:51:04.882734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:04.882702 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:04.882878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:04.882764 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:05.882238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:05.882207 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:05.882646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:05.882258 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:06.881885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:06.881846 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:06.882078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:06.881932 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:07.668226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:07.668188 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:07.882336 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:07.881910 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:07.882521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:07.882364 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:08.881550 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:08.881513 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:08.882091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:08.881580 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:09.571945 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:09.571917 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:51:09.881868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:09.881778 4494 patch_prober.go:29] interesting pod/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 02 23:51:09.881868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:09.881839 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 02 23:51:10.886999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:10.886970 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:51:10.888213 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:10.888164 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:51:12.987123 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:12.987084 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:17.025161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:17.025128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:17.568536 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:51:17.568489 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:51:18.249001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:18.248965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:23.547438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:23.547398 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:28.401900 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:28.401869 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:33.388999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:33.388963 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:38.238971 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.238934 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:38.903228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.903191 4494 generic.go:356] "Generic (PLEG): container finished" podID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerID="4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5" exitCode=0 Nov 02 23:51:38.903435 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.903202 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerDied","Data":"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5"} Nov 02 23:51:38.903682 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.903668 4494 scope.go:122] "RemoveContainer" containerID="4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5" Nov 02 23:51:38.905357 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.905327 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:38.905458 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.905407 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:38.905510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:38.905456 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:51:39.814529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:39.814493 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:39.912776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:39.912738 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerStarted","Data":"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c"} Nov 02 23:51:40.181052 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:40.181017 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:40.182925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:40.182908 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:51:40.183118 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:40.183093 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:40.183167 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:40.183136 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:41.183131 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:41.183094 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:41.183660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:41.183147 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:42.182918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:42.182878 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:42.183105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:42.182950 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:43.183387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:43.183346 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:43.183899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:43.183408 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:43.284873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:43.284841 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:44.182740 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:44.182706 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:44.182938 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:44.182767 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:45.182812 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:45.182772 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Nov 02 23:51:45.183236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:45.182830 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Nov 02 23:51:45.770112 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:45.770073 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:46.187043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:46.187005 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:51:46.187633 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:46.187422 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:51:46.188150 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:46.188133 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:51:46.584863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:46.584782 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:46.813326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:46.813296 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:51:47.563456 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:51:47.563242 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:51:47.621143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:47.621111 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:47.768915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:47.768875 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:47.913454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:47.913423 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:48.340819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:48.340715 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:48.487690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:48.487655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:49.740692 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:49.740650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:51.399984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:51.399942 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:52.248104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:52.248069 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:54.275889 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:54.275854 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:54.648186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:54.648068 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:57.503508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:57.503467 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:57.646475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:57.646438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:59.102399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:59.102356 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:59.637962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:59.637923 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:51:59.753748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:59.753719 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:51:59.753966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:59.753941 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" containerID="cri-o://e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e" gracePeriod=30 Nov 02 23:51:59.784596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:51:59.784564 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:01.616390 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:01.616352 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:02.244625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:02.244577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:02.412130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:02.412096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:04.856899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:04.856867 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:05.046225 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.046198 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:52:05.114920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.114881 4494 generic.go:356] "Generic (PLEG): container finished" podID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerID="e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e" exitCode=0 Nov 02 23:52:05.115109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.114946 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" Nov 02 23:52:05.115109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.114964 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerDied","Data":"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e"} Nov 02 23:52:05.115109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.115022 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9" event={"ID":"37bebd28-eaef-4ceb-84a3-b22b763b295c","Type":"ContainerDied","Data":"0d769a815699c2a2f48469886e993cd67d6c6ec0f630db3d84e52db4727f14ea"} Nov 02 23:52:05.115109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.115046 4494 scope.go:122] "RemoveContainer" containerID="e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e" Nov 02 23:52:05.125832 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.125806 4494 scope.go:122] "RemoveContainer" containerID="9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e" Nov 02 23:52:05.129164 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129144 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129202 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129261 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129248 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129291 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129341 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129429 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129442 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:05.129566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129466 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:05.129566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129516 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129640 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129645 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:05.129803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129672 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\") pod \"37bebd28-eaef-4ceb-84a3-b22b763b295c\" (UID: \"37bebd28-eaef-4ceb-84a3-b22b763b295c\") " Nov 02 23:52:05.129861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129817 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:05.129861 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.129831 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:52:05.130297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.130275 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.130403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.130313 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.130403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.130329 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.130403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.130340 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bebd28-eaef-4ceb-84a3-b22b763b295c-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.130403 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.130352 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.131930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.131900 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:52:05.132044 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.131972 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "kube-api-access-kqn9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:05.132129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.132108 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:05.132239 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.132223 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token" pod "37bebd28-eaef-4ceb-84a3-b22b763b295c" (UID: "37bebd28-eaef-4ceb-84a3-b22b763b295c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:05.136507 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.136485 4494 scope.go:122] "RemoveContainer" containerID="e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e" Nov 02 23:52:05.136819 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:05.136798 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e\": container with ID starting with e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e not found: ID does not exist" containerID="e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e" Nov 02 23:52:05.136915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.136825 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e"} err="failed to get container status \"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e\": rpc error: code = NotFound desc = could not find container \"e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e\": container with ID starting with e9d0d41e863def59200e92cfafaa669376adf785b33a5b061ea9369096ba3a9e not found: ID does not exist" Nov 02 23:52:05.136915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.136843 4494 scope.go:122] "RemoveContainer" containerID="9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e" Nov 02 23:52:05.137120 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:05.137098 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e\": container with ID starting with 9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e not found: ID does not exist" containerID="9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e" Nov 02 23:52:05.137284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.137125 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e"} err="failed to get container status \"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e\": rpc error: code = NotFound desc = could not find container \"9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e\": container with ID starting with 9a34c7bddb7395d618d5b53c0ab875831114e32df65c421a74503621f4d4ff9e not found: ID does not exist" Nov 02 23:52:05.231912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.231876 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.231912 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.231916 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.232124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.231931 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bebd28-eaef-4ceb-84a3-b22b763b295c-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.232124 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.231950 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqn9s\" (UniqueName: \"kubernetes.io/projected/37bebd28-eaef-4ceb-84a3-b22b763b295c-kube-api-access-kqn9s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:05.540496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.540457 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:52:05.545864 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.545840 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-d75a69-openshift-default-75647795fc-qv8v9"] Nov 02 23:52:05.695330 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:05.695247 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" path="/var/lib/kubelet/pods/37bebd28-eaef-4ceb-84a3-b22b763b295c/volumes" Nov 02 23:52:06.537783 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:06.537730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:06.725183 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:06.725126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:06.872814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:06.872736 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:08.581824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:08.581755 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:09.760845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:09.760808 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:09.933859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:09.933515 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:10.407244 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.397754 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:10.494360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.494328 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:52:10.494999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.494981 4494 cpu_manager.go:414] "containerMap: removing container" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.494999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.494995 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.495241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.495007 4494 cpu_manager.go:414] "containerMap: removing container" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.495241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.495012 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.495241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.495090 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.495241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.495100 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bebd28-eaef-4ceb-84a3-b22b763b295c" containerName="istio-proxy" Nov 02 23:52:10.500085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.500065 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.531431 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.531401 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.531577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.531534 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph44b\" (UniqueName: \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.555362 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.555328 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:52:10.601706 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.601676 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f7002-dockercfg-xlpjh\"" Nov 02 23:52:10.601875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.601712 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f7002\"" Nov 02 23:52:10.632606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.632574 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.632819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.632670 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph44b\" (UniqueName: \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.633387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.633359 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.694595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.694563 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph44b\" (UniqueName: \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\") pod \"payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:10.813882 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:10.813852 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:11.190824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:11.190799 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:52:11.191873 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:52:11.191849 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb746fac7_2995_41ea_b86a_75537acf4162.slice/crio-8093de9cd18ed121b93af81f90c0d466778d344cfdd5a6a1295a122f28bb0726 WatchSource:0}: Error finding container 8093de9cd18ed121b93af81f90c0d466778d344cfdd5a6a1295a122f28bb0726: Status 404 returned error can't find the container with id 8093de9cd18ed121b93af81f90c0d466778d344cfdd5a6a1295a122f28bb0726 Nov 02 23:52:12.165371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:12.165338 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" event={"ID":"b746fac7-2995-41ea-b86a-75537acf4162","Type":"ContainerStarted","Data":"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c"} Nov 02 23:52:12.165780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:12.165392 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" event={"ID":"b746fac7-2995-41ea-b86a-75537acf4162","Type":"ContainerStarted","Data":"8093de9cd18ed121b93af81f90c0d466778d344cfdd5a6a1295a122f28bb0726"} Nov 02 23:52:12.165780 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:12.165419 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:12.282944 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:12.282879 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" podStartSLOduration=2.282858708 podStartE2EDuration="2.282858708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:12.281549518 +0000 UTC m=+3846.711419379" watchObservedRunningTime="2026-11-02 23:52:12.282858708 +0000 UTC m=+3846.712728569" Nov 02 23:52:14.058625 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:14.058594 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:14.206396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:14.206367 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:14.612592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:14.612556 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:15.490739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:15.490703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:17.368564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:17.368528 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:17.559808 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:17.559770 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:52:20.147701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:20.147662 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:20.879473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:20.879439 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:23.174672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:23.174640 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:52:26.259446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:26.259411 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:28.718696 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:28.718664 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:52:31.628492 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:31.628455 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:36.498184 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:36.498135 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:41.437756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:41.437724 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:45.653536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:45.653505 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:52:45.654058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:45.653694 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" podUID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" containerName="maas-api" containerID="cri-o://24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0" gracePeriod=30 Nov 02 23:52:45.755676 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:45.755646 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:45.954127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:45.954102 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:52:46.101191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.094597 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\" (UniqueName: \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\") pod \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " Nov 02 23:52:46.101191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.094662 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") pod \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " Nov 02 23:52:46.101191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.094738 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\") pod \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " Nov 02 23:52:46.101191 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.094952 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\") pod \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\" (UID: \"cce4c8b1-ceb4-46c4-ab52-dc547651ed66\") " Nov 02 23:52:46.117197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.113460 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca" pod "cce4c8b1-ceb4-46c4-ab52-dc547651ed66" (UID: "cce4c8b1-ceb4-46c4-ab52-dc547651ed66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:52:46.117529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.117336 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs" pod "cce4c8b1-ceb4-46c4-ab52-dc547651ed66" (UID: "cce4c8b1-ceb4-46c4-ab52-dc547651ed66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:52:46.119638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.119606 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl" pod "cce4c8b1-ceb4-46c4-ab52-dc547651ed66" (UID: "cce4c8b1-ceb4-46c4-ab52-dc547651ed66"). InnerVolumeSpecName "kube-api-access-5z9zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:46.121899 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.121850 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls" pod "cce4c8b1-ceb4-46c4-ab52-dc547651ed66" (UID: "cce4c8b1-ceb4-46c4-ab52-dc547651ed66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:52:46.219726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.218499 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:46.219726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.218539 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:46.219726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.218557 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9zl\" (UniqueName: \"kubernetes.io/projected/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-kube-api-access-5z9zl\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:46.219726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.218574 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cce4c8b1-ceb4-46c4-ab52-dc547651ed66-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:46.407776 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.407743 4494 generic.go:356] "Generic (PLEG): container finished" podID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" containerID="24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0" exitCode=0 Nov 02 23:52:46.407958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.407797 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" Nov 02 23:52:46.407958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.407834 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" event={"ID":"cce4c8b1-ceb4-46c4-ab52-dc547651ed66","Type":"ContainerDied","Data":"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0"} Nov 02 23:52:46.407958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.407888 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp" event={"ID":"cce4c8b1-ceb4-46c4-ab52-dc547651ed66","Type":"ContainerDied","Data":"982f6fee18ec5311628b2cfd1cf6deb784d89586637ffb549135fef29dc50c00"} Nov 02 23:52:46.407958 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.407910 4494 scope.go:122] "RemoveContainer" containerID="24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0" Nov 02 23:52:46.418809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.418792 4494 scope.go:122] "RemoveContainer" containerID="24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0" Nov 02 23:52:46.419073 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:46.419053 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0\": container with ID starting with 24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0 not found: ID does not exist" containerID="24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0" Nov 02 23:52:46.419143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.419081 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0"} err="failed to get container status \"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0\": rpc error: code = NotFound desc = could not find container \"24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0\": container with ID starting with 24b74f525de8d88fc7590a4bb9c2f70eb525d6b2e52e4174d053de6dcd8d84f0 not found: ID does not exist" Nov 02 23:52:46.576524 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.576441 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:52:46.580259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:46.580231 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0f49ab-56fb7cb986-4vdjp"] Nov 02 23:52:47.459746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.459706 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:47.598596 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:47.598543 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:52:47.604331 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.604306 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:47.694681 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.694649 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" path="/var/lib/kubelet/pods/cce4c8b1-ceb4-46c4-ab52-dc547651ed66/volumes" Nov 02 23:52:47.943677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.943646 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:47.947428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.947396 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:52:47.947672 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:47.947648 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" containerID="cri-o://d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e" gracePeriod=30 Nov 02 23:52:51.529154 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:51.529107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:51.677401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:51.677368 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:52.784828 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:52.784783 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:52.934611 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:52.934574 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:53.211613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.211591 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:52:53.235733 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235692 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.235875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235759 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.235875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235817 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.235875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235863 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235882 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235903 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235931 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.235977 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236018 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236046 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:53.236488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236126 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\") pod \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\" (UID: \"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268\") " Nov 02 23:52:53.236488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236135 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:53.236488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236198 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:53.236488 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236316 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:52:53.236665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236631 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.236665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236652 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.236750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236667 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.236750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236681 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.236750 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.236695 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.238382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.238356 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "kube-api-access-bqhrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:53.238770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.238740 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:52:53.238770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.238751 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:52:53.238908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.238804 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token" pod "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" (UID: "8bc39ed6-e3c7-46c6-a2fe-fdba708bc268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:52:53.337876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.337790 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhrt\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-kube-api-access-bqhrt\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.337876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.337826 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.337876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.337839 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.337876 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.337851 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:52:53.457386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.457355 4494 generic.go:356] "Generic (PLEG): container finished" podID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerID="d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e" exitCode=0 Nov 02 23:52:53.457575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.457431 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" Nov 02 23:52:53.457575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.457452 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerDied","Data":"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e"} Nov 02 23:52:53.457575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.457509 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc" event={"ID":"8bc39ed6-e3c7-46c6-a2fe-fdba708bc268","Type":"ContainerDied","Data":"95c09ad0596d4924fe9f37bc128d0c2936e0d9113d933a8dcebacef66215ce31"} Nov 02 23:52:53.457575 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.457531 4494 scope.go:122] "RemoveContainer" containerID="d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e" Nov 02 23:52:53.468585 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.468569 4494 scope.go:122] "RemoveContainer" containerID="696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0" Nov 02 23:52:53.478132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.478109 4494 scope.go:122] "RemoveContainer" containerID="d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e" Nov 02 23:52:53.478422 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:53.478401 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e\": container with ID starting with d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e not found: ID does not exist" containerID="d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e" Nov 02 23:52:53.478493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.478431 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e"} err="failed to get container status \"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e\": rpc error: code = NotFound desc = could not find container \"d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e\": container with ID starting with d75c827bb7ea05c032adbbe12130294dbddf0bf271e7ab00a31765e09d3d2d1e not found: ID does not exist" Nov 02 23:52:53.478493 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.478449 4494 scope.go:122] "RemoveContainer" containerID="696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0" Nov 02 23:52:53.478699 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:52:53.478681 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0\": container with ID starting with 696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0 not found: ID does not exist" containerID="696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0" Nov 02 23:52:53.478743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.478704 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0"} err="failed to get container status \"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0\": rpc error: code = NotFound desc = could not find container \"696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0\": container with ID starting with 696db8d22acb7978eee10e9ee0d895ba176d7070078e72a45e6f63ebfcadb1b0 not found: ID does not exist" Nov 02 23:52:53.624533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.624454 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:52:53.629919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.629895 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f49ab-openshift-default-5bcbd8c97c-mlgsc"] Nov 02 23:52:53.695714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:53.695685 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" path="/var/lib/kubelet/pods/8bc39ed6-e3c7-46c6-a2fe-fdba708bc268/volumes" Nov 02 23:52:55.176422 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:55.176381 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:55.322635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:55.322595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:55.811059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:55.811017 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:52:57.906964 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:52:57.906919 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:02.095486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:02.095438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:03.859910 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:03.859873 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:07.687197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:07.687125 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:07.831769 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:07.831739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:08.866902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:08.866861 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:09.177096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:09.177063 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:09.643328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:09.643274 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:09.646717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:09.646692 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:53:09.646925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:09.646906 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" containerID="cri-o://a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae" gracePeriod=30 Nov 02 23:53:10.390995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:10.390958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:14.285829 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:14.285801 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:14.911236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:14.911146 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:53:15.027475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027445 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027566 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027615 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027635 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027681 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027712 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027734 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.027807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027748 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:15.028036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027823 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.028036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027885 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\") pod \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\" (UID: \"50f41f69-f9c2-4339-8417-c0f26f5d4ad9\") " Nov 02 23:53:15.028036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027902 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:15.028036 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.027916 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:15.028304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028218 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:15.028304 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028256 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:53:15.028683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028549 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.028771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028693 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.028771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028710 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.028771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028723 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.028771 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.028738 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.030096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.030068 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "kube-api-access-4jpgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:15.030272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.030087 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:53:15.030272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.030140 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:53:15.030371 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.030344 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo" pod "50f41f69-f9c2-4339-8417-c0f26f5d4ad9" (UID: "50f41f69-f9c2-4339-8417-c0f26f5d4ad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:53:15.130434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.130394 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jpgj\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-kube-api-access-4jpgj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.130434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.130429 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.130434 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.130442 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.130670 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.130454 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f41f69-f9c2-4339-8417-c0f26f5d4ad9-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:53:15.613241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.613209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerDied","Data":"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae"} Nov 02 23:53:15.613741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.613264 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" Nov 02 23:53:15.613741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.613284 4494 scope.go:122] "RemoveContainer" containerID="a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae" Nov 02 23:53:15.613741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.613203 4494 generic.go:356] "Generic (PLEG): container finished" podID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerID="a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae" exitCode=0 Nov 02 23:53:15.613741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.613412 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb" event={"ID":"50f41f69-f9c2-4339-8417-c0f26f5d4ad9","Type":"ContainerDied","Data":"2111f0a9ced83b5aaada64e477de64a902ba297c0329def4e05d9935bfb708a8"} Nov 02 23:53:15.624960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.624943 4494 scope.go:122] "RemoveContainer" containerID="7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661" Nov 02 23:53:15.634376 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.634361 4494 scope.go:122] "RemoveContainer" containerID="a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae" Nov 02 23:53:15.634650 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:53:15.634631 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae\": container with ID starting with a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae not found: ID does not exist" containerID="a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae" Nov 02 23:53:15.634697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.634657 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae"} err="failed to get container status \"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae\": rpc error: code = NotFound desc = could not find container \"a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae\": container with ID starting with a8fe04f4a13b3e2096754319b4cd5e57542be32f139bfc2a4504ee835508deae not found: ID does not exist" Nov 02 23:53:15.634697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.634675 4494 scope.go:122] "RemoveContainer" containerID="7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661" Nov 02 23:53:15.634933 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:53:15.634913 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661\": container with ID starting with 7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661 not found: ID does not exist" containerID="7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661" Nov 02 23:53:15.634979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.634940 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661"} err="failed to get container status \"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661\": rpc error: code = NotFound desc = could not find container \"7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661\": container with ID starting with 7c5abf96914f7205b4bb71ca1e353de53758f1fc166e37c2c22d12020ebbe661 not found: ID does not exist" Nov 02 23:53:15.835214 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.835136 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:53:15.838751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.838725 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3058ee-openshift-default-fdf4b56df-gs4pb"] Nov 02 23:53:15.857206 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:15.857184 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:16.001411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:16.001371 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:16.433284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:16.433246 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:17.563906 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:53:17.563813 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:53:17.695659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:17.695617 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" path="/var/lib/kubelet/pods/50f41f69-f9c2-4339-8417-c0f26f5d4ad9/volumes" Nov 02 23:53:17.729560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:17.729527 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:19.578048 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:19.578010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:22.072070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:22.072038 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:24.740543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:24.740507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:25.150603 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:25.150527 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:53:29.096838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:29.096795 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:30.698068 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:30.698038 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:30.852995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:30.852960 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:31.001021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:31.000939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:35.728382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:35.728349 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:37.942095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:37.942059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:38.087553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:38.087520 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:39.612091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:39.612057 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:40.126040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:40.126005 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:40.877849 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:40.877815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:41.026907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:41.026849 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:46.051265 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:46.051224 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:46.330039 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:46.329957 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:46.473109 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:46.473070 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:47.071063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:47.071030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:47.561364 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:53:47.561326 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:53:50.365404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:50.365369 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:50.532483 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:50.532444 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:50.675059 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:50.675009 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:51.244413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:51.244375 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:52.782594 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:52.782558 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:52.923915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:52.923863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:53.419079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:53.419038 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:56.237843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:56.237804 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:56.577312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:56.577114 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:58.792405 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.792364 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 02 23:53:58.793221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793193 4494 cpu_manager.go:414] "containerMap: removing container" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793218 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793250 4494 cpu_manager.go:414] "containerMap: removing container" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793255 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793268 4494 cpu_manager.go:414] "containerMap: removing container" podUID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" containerName="maas-api" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793274 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" containerName="maas-api" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793282 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793287 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793304 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.793401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793309 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.793658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793425 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793436 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f41f69-f9c2-4339-8417-c0f26f5d4ad9" containerName="istio-proxy" Nov 02 23:53:58.793658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793443 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce4c8b1-ceb4-46c4-ab52-dc547651ed66" containerName="maas-api" Nov 02 23:53:58.793658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793450 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.793658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.793457 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc39ed6-e3c7-46c6-a2fe-fdba708bc268" containerName="istio-proxy" Nov 02 23:53:58.797025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.797007 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.864536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.864488 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 02 23:53:58.918684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.918654 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 02 23:53:58.969156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969122 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969161 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969278 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrcv\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969312 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969341 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969406 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969458 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969513 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:58.969626 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:58.969556 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.055401 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.055316 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:59.071248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071221 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071286 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071307 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071358 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrcv\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071393 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071385 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071425 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071478 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071538 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071601 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071597 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071816 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.071852 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.071834 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.072116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.072096 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.072252 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.072160 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.072820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.072796 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.073851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.073828 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.074145 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.074127 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.132895 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.132858 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.133283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.133260 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrcv\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\") pod \"maas-default-gateway-openshift-default-694db55c87-ql99p\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.199874 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.199831 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:53:59.411628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.411598 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:53:59.841967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.841944 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 02 23:53:59.842910 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:53:59.842887 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159a248d_7ffa_4fa5_b2f6_11690ecac868.slice/crio-317de9bfa61169ff4f1adfe89f6dabbe52b012958826cad6c1fd6b35444ff201 WatchSource:0}: Error finding container 317de9bfa61169ff4f1adfe89f6dabbe52b012958826cad6c1fd6b35444ff201: Status 404 returned error can't find the container with id 317de9bfa61169ff4f1adfe89f6dabbe52b012958826cad6c1fd6b35444ff201 Nov 02 23:53:59.845807 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.845769 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.845884 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.845857 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.845951 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.845914 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:53:59.917843 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.917809 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" event={"ID":"159a248d-7ffa-4fa5-b2f6-11690ecac868","Type":"ContainerStarted","Data":"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525"} Nov 02 23:53:59.918001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:53:59.917864 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" event={"ID":"159a248d-7ffa-4fa5-b2f6-11690ecac868","Type":"ContainerStarted","Data":"317de9bfa61169ff4f1adfe89f6dabbe52b012958826cad6c1fd6b35444ff201"} Nov 02 23:54:00.042810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:00.042751 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podStartSLOduration=2.042733091 podStartE2EDuration="2.042733091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:00.0399356 +0000 UTC m=+3954.469805460" watchObservedRunningTime="2026-11-02 23:54:00.042733091 +0000 UTC m=+3954.472602954" Nov 02 23:54:00.412457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:00.412428 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:54:00.413772 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:00.413747 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:00.413935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:00.413803 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:00.614205 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:00.614088 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:01.345747 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:01.345710 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:01.414616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:01.414579 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:01.414966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:01.414928 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:02.412105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:02.412068 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:02.412566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:02.412123 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:03.412024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:03.411987 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:03.412253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:03.412042 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:04.412193 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:04.412139 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:04.412375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:04.412223 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:05.156701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:05.156658 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:05.305294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:05.305256 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:05.412743 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:05.412708 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:05.413272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:05.412769 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:06.411966 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:06.411928 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:06.412160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:06.411999 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:06.483229 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:06.483199 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:06.930063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:06.930032 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:07.412182 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:07.412143 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Nov 02 23:54:07.412349 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:07.412222 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Nov 02 23:54:09.412990 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:09.412938 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:54:09.413508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:09.413033 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:54:10.455386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:10.455342 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:54:10.456128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:10.456082 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:54:11.497386 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.497340 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ql99p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:54:11.497891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.497434 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:54:11.537933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.537885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:11.602273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.602237 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:54:11.602768 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.602749 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:54:11.603559 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.603542 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 02 23:54:11.689104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.689021 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:11.834704 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.834594 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:11.977419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:11.977386 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:13.794444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:13.794407 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:13.944316 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:13.944279 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:16.459237 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:16.459200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:17.560161 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:54:17.560125 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:54:18.076553 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:18.076515 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:20.136381 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:20.136347 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:20.280199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:20.280155 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:20.441196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:20.433831 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:21.395770 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:21.395706 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:21.691050 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:21.690995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:21.845394 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:21.845358 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:23.609605 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:23.609562 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:24.115833 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.115802 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:54:24.119803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.119786 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.184196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.183709 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-18a6d8-openshift-default-dockercfg-gtrmj\"" Nov 02 23:54:24.241196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.241125 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:24.316400 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.316340 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:54:24.319218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.319188 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.319456 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.319436 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.319582 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.319569 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.319703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.319689 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.319902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.319796 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.320084 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.320070 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.320298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.320280 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9vfq\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.320450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.320437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.320566 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.320553 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.421909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.421877 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.421939 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422012 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422103 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422061 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422106 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422207 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422243 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422360 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422328 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422401 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422430 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422637 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422499 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9vfq\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422834 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.422870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422856 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.423069 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.422914 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.424855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.424831 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.424954 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.424936 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.483797 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.483764 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9vfq\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.483949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.483843 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\") pod \"e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.495762 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.495734 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:24.531415 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.529785 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:24.962628 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.962599 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:54:24.962951 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:54:24.962655 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61531ff1_458f_420a_92ed_8df67e3eb4bb.slice/crio-892c130d6f34390d7a1dc3b37c871842a55911816464026a10f5ac93e3cdb74f WatchSource:0}: Error finding container 892c130d6f34390d7a1dc3b37c871842a55911816464026a10f5ac93e3cdb74f: Status 404 returned error can't find the container with id 892c130d6f34390d7a1dc3b37c871842a55911816464026a10f5ac93e3cdb74f Nov 02 23:54:24.965367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.965334 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:24.965472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.965401 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:24.965472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:24.965430 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:25.229521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:25.229428 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerStarted","Data":"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499"} Nov 02 23:54:25.229521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:25.229504 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerStarted","Data":"892c130d6f34390d7a1dc3b37c871842a55911816464026a10f5ac93e3cdb74f"} Nov 02 23:54:25.354404 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:25.354355 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podStartSLOduration=1.3543347159999999 podStartE2EDuration="1.354334716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:25.352614223 +0000 UTC m=+3979.782484109" watchObservedRunningTime="2026-11-02 23:54:25.354334716 +0000 UTC m=+3979.784204577" Nov 02 23:54:25.504355 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:25.504270 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:26.326504 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:26.326472 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:26.502577 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:26.502548 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:27.242517 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:27.242488 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:27.243642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:27.243622 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:54:28.091238 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:28.091145 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:28.267819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:28.267782 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:28.412129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:28.412102 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:28.700627 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:28.700544 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:54:31.294982 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:31.294950 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:31.483198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:31.482880 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:31.702819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:31.702793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:36.012604 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:36.012572 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:37.748477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:37.748431 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:41.878387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:41.878353 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:42.344240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:42.344183 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:42.491865 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:42.491791 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:42.752079 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:42.752046 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:43.169337 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.169304 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:43.667906 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.667871 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 02 23:54:43.672339 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.672321 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.736199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.736095 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5db41e-openshift-default-dockercfg-hqkt8\"" Nov 02 23:54:43.871188 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.853963 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:43.871775 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.856326 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 02 23:54:43.890646 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890613 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.890892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890699 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.890892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890760 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.890892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890802 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhrf\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.890892 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890876 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.891160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.890931 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.891160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.891021 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.891296 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.891272 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.891359 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.891329 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992305 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992364 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992434 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992583 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992657 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992625 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992657 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992689 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992712 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992729 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.992841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992781 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhrf\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.993130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.992975 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.993279 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.993243 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.993426 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.993246 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.993484 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.993358 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.995096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.995074 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:43.995450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:43.995423 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:44.054593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.054555 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:44.054731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.054715 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhrf\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\") pod \"e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:44.161573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.158543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:44.165531 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.165498 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 02 23:54:44.172141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.172114 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.260309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.260221 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 02 23:54:44.295933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.295903 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:44.296739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.296702 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.296902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.296803 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.296902 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.296858 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.296934 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.297068 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.297127 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9vm\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.297198 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.297242 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.297289 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.297269 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398519 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398567 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398686 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398752 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9vm\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398781 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398829 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398861 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398911 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.398958 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.399725 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.399637 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.401161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.399836 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.401161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.399925 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.401161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.400048 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.401161 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.400561 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.402887 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.402851 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.403083 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.403019 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.460825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.460796 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.460825 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.460816 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9vm\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\") pod \"maas-default-gateway-openshift-default-694db55c87-5rkg8\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.488058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.488033 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:44.741994 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.741963 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 02 23:54:44.742668 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:54:44.742646 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4ea97d_9907_4240_9f3d_94f7d7f92174.slice/crio-9c61682a597a3bd98d77772b162407545ffc9c1bfdd35fcc8db17295c6e9a791 WatchSource:0}: Error finding container 9c61682a597a3bd98d77772b162407545ffc9c1bfdd35fcc8db17295c6e9a791: Status 404 returned error can't find the container with id 9c61682a597a3bd98d77772b162407545ffc9c1bfdd35fcc8db17295c6e9a791 Nov 02 23:54:44.745647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.745614 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.745739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.745677 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.745739 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.745705 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.882352 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.882326 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 02 23:54:44.883473 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:54:44.883448 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e32ea6_02e2_4808_ac8e_f79f2948cfeb.slice/crio-5e4814d544021c3b52755b67b1f2fcd66021b110d3826b1f7cd353c0c1306a79 WatchSource:0}: Error finding container 5e4814d544021c3b52755b67b1f2fcd66021b110d3826b1f7cd353c0c1306a79: Status 404 returned error can't find the container with id 5e4814d544021c3b52755b67b1f2fcd66021b110d3826b1f7cd353c0c1306a79 Nov 02 23:54:44.887104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.887069 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.887266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.887141 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:44.887266 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:44.887206 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:54:45.410312 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.410274 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerStarted","Data":"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3"} Nov 02 23:54:45.410791 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.410347 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerStarted","Data":"9c61682a597a3bd98d77772b162407545ffc9c1bfdd35fcc8db17295c6e9a791"} Nov 02 23:54:45.412321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.412290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" event={"ID":"55e32ea6-02e2-4808-ac8e-f79f2948cfeb","Type":"ContainerStarted","Data":"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32"} Nov 02 23:54:45.412471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.412339 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" event={"ID":"55e32ea6-02e2-4808-ac8e-f79f2948cfeb","Type":"ContainerStarted","Data":"5e4814d544021c3b52755b67b1f2fcd66021b110d3826b1f7cd353c0c1306a79"} Nov 02 23:54:45.489200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.489137 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:45.490988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.490957 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:45.491209 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.491009 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:45.535021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:45.534963 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podStartSLOduration=2.534944943 podStartE2EDuration="2.534944943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:45.534483232 +0000 UTC m=+3999.964353092" watchObservedRunningTime="2026-11-02 23:54:45.534944943 +0000 UTC m=+3999.964814804" Nov 02 23:54:46.296753 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.296718 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:46.302969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.302945 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:46.419343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.419308 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:46.420399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.420377 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:54:46.429512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.429467 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podStartSLOduration=2.429453847 podStartE2EDuration="2.429453847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:54:45.69594165 +0000 UTC m=+4000.125811518" watchObservedRunningTime="2026-11-02 23:54:46.429453847 +0000 UTC m=+4000.859323705" Nov 02 23:54:46.488996 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.488954 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:46.489201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:46.489034 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:47.488901 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:47.488864 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:47.489361 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:47.488913 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:47.594740 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:54:47.594653 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:54:48.260267 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:48.260215 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:48.407618 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:48.407589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:48.489051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:48.489016 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:48.489498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:48.489068 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:48.563078 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:48.562985 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:49.489097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:49.489065 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:49.489549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:49.489118 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:50.489512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:50.489481 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:50.490021 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:50.489535 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:51.488514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:51.488482 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:51.488873 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:51.488535 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:51.573038 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:51.572989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:51.773542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:51.772287 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:52.489298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:52.489263 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 02 23:54:52.489490 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:52.489313 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 02 23:54:53.674714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:53.674671 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:54:53.883202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:53.883148 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:53.883693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:53.883667 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:53.884752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:53.884735 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 02 23:54:58.755129 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:54:58.755083 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:00.444927 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:00.444890 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:01.822290 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:01.822256 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:02.375572 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:02.375447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:02.752236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:02.752163 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:03.955055 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:03.955020 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:06.833980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:06.833904 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:06.998978 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:06.998946 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:07.144659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:07.144577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:08.325717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:08.325672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:08.536090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:08.536052 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:55:08.536335 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:08.536311 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" podUID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" containerName="maas-api" containerID="cri-o://ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27" gracePeriod=30 Nov 02 23:55:08.641406 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:08.641321 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:08.856367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:08.856342 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:55:09.022471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.022245 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\") pod \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " Nov 02 23:55:09.022471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.022405 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") pod \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " Nov 02 23:55:09.022471 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.022457 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\") pod \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " Nov 02 23:55:09.022822 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.022551 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\" (UniqueName: \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\") pod \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\" (UID: \"9db42416-5e58-4da3-b6c5-d8393d0a97e9\") " Nov 02 23:55:09.024773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.023632 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca" pod "9db42416-5e58-4da3-b6c5-d8393d0a97e9" (UID: "9db42416-5e58-4da3-b6c5-d8393d0a97e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:55:09.030476 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.030448 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf" pod "9db42416-5e58-4da3-b6c5-d8393d0a97e9" (UID: "9db42416-5e58-4da3-b6c5-d8393d0a97e9"). InnerVolumeSpecName "kube-api-access-d7mkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:09.034520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.034487 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs" pod "9db42416-5e58-4da3-b6c5-d8393d0a97e9" (UID: "9db42416-5e58-4da3-b6c5-d8393d0a97e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:55:09.036593 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.036566 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls" pod "9db42416-5e58-4da3-b6c5-d8393d0a97e9" (UID: "9db42416-5e58-4da3-b6c5-d8393d0a97e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:55:09.124908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.124725 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:09.124908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.124761 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db42416-5e58-4da3-b6c5-d8393d0a97e9-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:09.124908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.124781 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db42416-5e58-4da3-b6c5-d8393d0a97e9-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:09.124908 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.124811 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mkf\" (UniqueName: \"kubernetes.io/projected/9db42416-5e58-4da3-b6c5-d8393d0a97e9-kube-api-access-d7mkf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:09.645720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.644132 4494 generic.go:356] "Generic (PLEG): container finished" podID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" containerID="ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27" exitCode=0 Nov 02 23:55:09.645720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.644229 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" event={"ID":"9db42416-5e58-4da3-b6c5-d8393d0a97e9","Type":"ContainerDied","Data":"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27"} Nov 02 23:55:09.645720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.644315 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" event={"ID":"9db42416-5e58-4da3-b6c5-d8393d0a97e9","Type":"ContainerDied","Data":"d51bc56108d712005295f4b73770f6a1859bb4bb69fb239522978b3300fb8cfd"} Nov 02 23:55:09.645720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.644351 4494 scope.go:122] "RemoveContainer" containerID="ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27" Nov 02 23:55:09.645720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.644508 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq" Nov 02 23:55:09.664508 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.664483 4494 scope.go:122] "RemoveContainer" containerID="ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27" Nov 02 23:55:09.664886 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:55:09.664864 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27\": container with ID starting with ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27 not found: ID does not exist" containerID="ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27" Nov 02 23:55:09.664969 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.664899 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27"} err="failed to get container status \"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27\": rpc error: code = NotFound desc = could not find container \"ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27\": container with ID starting with ea6555784c13a8d398e455b8e3c360025194c37c9d8bfed365e9d0126416be27 not found: ID does not exist" Nov 02 23:55:09.813080 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.813042 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:55:09.863063 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:09.863031 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-056584-75cc65df45-qdksq"] Nov 02 23:55:10.556690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:10.556657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:11.694272 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:11.694241 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" path="/var/lib/kubelet/pods/9db42416-5e58-4da3-b6c5-d8393d0a97e9/volumes" Nov 02 23:55:12.002859 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.002784 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:12.162867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.162837 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:12.383695 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.383615 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:12.851104 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.851069 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:55:12.851568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.851337 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" containerID="cri-o://4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c" gracePeriod=30 Nov 02 23:55:12.865515 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:12.865476 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:13.655085 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:13.655059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:13.807665 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:13.807576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:13.971530 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:13.971482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:14.522501 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.522470 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:55:14.523106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.523092 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" containerName="maas-api" Nov 02 23:55:14.523163 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.523108 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" containerName="maas-api" Nov 02 23:55:14.523248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.523239 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db42416-5e58-4da3-b6c5-d8393d0a97e9" containerName="maas-api" Nov 02 23:55:14.526529 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.526509 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565201 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565145 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565202 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565351 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565316 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565353 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565435 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565648 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565625 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmrdg\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565697 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565681 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.565814 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.565794 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.591475 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.591451 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7097dd-openshift-default-dockercfg-h5lfd\"" Nov 02 23:55:14.636162 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.636130 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667628 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667714 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667747 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667835 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667878 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.667940 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668019 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668085 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmrdg\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668120 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668244 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668489 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.668822 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.670144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.669329 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.671382 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.671253 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.671980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.671952 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.729755 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.729710 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.729915 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.729807 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmrdg\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\") pod \"e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:14.768845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.768806 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:14.841043 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:14.840963 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:15.051113 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.051073 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:15.276051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.276012 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:55:15.276776 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:55:15.276747 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64ead7f_e1af_4e13_ac03_faef9ddf6cfa.slice/crio-b6304cd50245768993e282e400815b9c3fc7c352c08f38a07da47cd7c5ff7682 WatchSource:0}: Error finding container b6304cd50245768993e282e400815b9c3fc7c352c08f38a07da47cd7c5ff7682: Status 404 returned error can't find the container with id b6304cd50245768993e282e400815b9c3fc7c352c08f38a07da47cd7c5ff7682 Nov 02 23:55:15.279587 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.279559 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:15.279662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.279622 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:15.279662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.279653 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:15.697082 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.697046 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerStarted","Data":"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4"} Nov 02 23:55:15.697288 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.697099 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerStarted","Data":"b6304cd50245768993e282e400815b9c3fc7c352c08f38a07da47cd7c5ff7682"} Nov 02 23:55:15.814510 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.814455 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podStartSLOduration=1.8144413849999999 podStartE2EDuration="1.814441385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:15.812997134 +0000 UTC m=+4030.242867009" watchObservedRunningTime="2026-11-02 23:55:15.814441385 +0000 UTC m=+4030.244311257" Nov 02 23:55:15.841679 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.841650 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:15.976957 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:15.976857 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:16.190363 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:16.190326 4494 patch_prober.go:29] interesting pod/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:55:16.190752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:16.190381 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:55:16.848250 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:16.848219 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:17.561808 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:55:17.561766 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:55:17.772198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:17.771596 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:17.787792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:17.785508 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:55:18.148715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.148691 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:55:18.238647 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238609 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.238869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238692 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.238869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238776 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.238869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238805 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.238869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238842 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.239091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238894 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.239091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238953 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:18.239091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.238995 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.239091 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239038 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.239341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239135 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\") pod \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\" (UID: \"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6\") " Nov 02 23:55:18.239399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239277 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:55:18.239399 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239375 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:18.239498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239390 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:18.239795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239766 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.239795 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239794 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.240012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239805 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.240012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239813 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.240012 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.239774 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:18.241271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.241247 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:55:18.241271 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.241256 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:18.241428 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.241247 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:55:18.241732 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.241713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk" pod "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" (UID: "d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6"). InnerVolumeSpecName "kube-api-access-gzwfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:18.341496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.341456 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.341496 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.341494 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.341717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.341509 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.341717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.341525 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.341717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.341540 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwfk\" (UniqueName: \"kubernetes.io/projected/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6-kube-api-access-gzwfk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:55:18.449218 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.449150 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:18.634563 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.634527 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:18.763810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.763714 4494 generic.go:356] "Generic (PLEG): container finished" podID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerID="4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c" exitCode=0 Nov 02 23:55:18.763810 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.763759 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerDied","Data":"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c"} Nov 02 23:55:18.764031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.763829 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" event={"ID":"d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6","Type":"ContainerDied","Data":"7997c47b1b63e92ffe9fa44a272521d40693452c7c69aab70eafd483343b9f76"} Nov 02 23:55:18.764031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.763776 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5" Nov 02 23:55:18.764031 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.763864 4494 scope.go:122] "RemoveContainer" containerID="4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c" Nov 02 23:55:18.780703 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.780674 4494 scope.go:122] "RemoveContainer" containerID="4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5" Nov 02 23:55:18.791738 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.791718 4494 scope.go:122] "RemoveContainer" containerID="4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c" Nov 02 23:55:18.792262 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:55:18.792235 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c\": container with ID starting with 4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c not found: ID does not exist" containerID="4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c" Nov 02 23:55:18.792375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.792274 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c"} err="failed to get container status \"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c\": rpc error: code = NotFound desc = could not find container \"4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c\": container with ID starting with 4feb5e42f8d922e2629ada0d9f3bf4b427d06590ef940e83d9cd7466e5937c6c not found: ID does not exist" Nov 02 23:55:18.792375 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.792299 4494 scope.go:122] "RemoveContainer" containerID="4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5" Nov 02 23:55:18.792856 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:55:18.792828 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5\": container with ID starting with 4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5 not found: ID does not exist" containerID="4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5" Nov 02 23:55:18.792962 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.792868 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5"} err="failed to get container status \"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5\": rpc error: code = NotFound desc = could not find container \"4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5\": container with ID starting with 4b723d86a0c9373057039e79dfbb399df4a7f119ff0f502a7ff25b07075583a5 not found: ID does not exist" Nov 02 23:55:18.841684 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.841652 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:18.934760 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.934720 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:55:18.985338 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.985298 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-056584-openshift-default-7db858c669-mfqd5"] Nov 02 23:55:18.996878 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:18.996843 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:19.699512 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:19.699475 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" path="/var/lib/kubelet/pods/d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6/volumes" Nov 02 23:55:21.473623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:21.473589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:21.892411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:21.892317 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:22.076046 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:22.076009 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:26.774521 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:26.774482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:29.511283 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.511245 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 02 23:55:29.512128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.512105 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:29.512128 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.512127 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:29.512332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.512186 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:29.512332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.512195 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:29.512444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.512338 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:29.516122 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.516101 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529058 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529029 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529241 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529221 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529317 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529268 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529397 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529380 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529413 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529464 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529489 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529510 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wq5\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.529636 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.529623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.623217 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.623165 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 02 23:55:29.631307 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631279 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631335 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631418 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631477 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631444 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631508 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631544 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631583 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631609 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631666 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631634 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wq5\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631747 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.631950 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.631848 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.632090 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.632067 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.632166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.632091 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.632166 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.632122 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.636392 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.636372 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.636730 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.636709 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.693127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.693098 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.693324 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.693305 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wq5\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\") pod \"maas-default-gateway-openshift-default-694db55c87-6dlpr\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:29.829870 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:29.829793 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:30.261325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.261295 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 02 23:55:30.262920 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:55:30.262894 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd07ebce_72d2_4d4a_bc36_629c06d07fde.slice/crio-d8ba85fb2afde49b7bace8a3c3c081be6405ac682f1abf91db73ad61b108f6cf WatchSource:0}: Error finding container d8ba85fb2afde49b7bace8a3c3c081be6405ac682f1abf91db73ad61b108f6cf: Status 404 returned error can't find the container with id d8ba85fb2afde49b7bace8a3c3c081be6405ac682f1abf91db73ad61b108f6cf Nov 02 23:55:30.265586 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.265555 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:30.265690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.265636 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:30.265690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.265669 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:30.876907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.876865 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" event={"ID":"bd07ebce-72d2-4d4a-bc36-629c06d07fde","Type":"ContainerStarted","Data":"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616"} Nov 02 23:55:30.877320 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.876959 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" event={"ID":"bd07ebce-72d2-4d4a-bc36-629c06d07fde","Type":"ContainerStarted","Data":"d8ba85fb2afde49b7bace8a3c3c081be6405ac682f1abf91db73ad61b108f6cf"} Nov 02 23:55:30.998939 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:30.998885 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podStartSLOduration=1.998872502 podStartE2EDuration="1.998872502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:30.997953533 +0000 UTC m=+4045.427823395" watchObservedRunningTime="2026-11-02 23:55:30.998872502 +0000 UTC m=+4045.428742363" Nov 02 23:55:31.851980 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:31.847550 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:31.856443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:31.856361 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:31.856443 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:31.856430 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:31.883369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:31.883335 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:32.851196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:32.848110 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:32.851196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:32.848242 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:33.853197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:33.852285 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:33.853197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:33.852362 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:34.830720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:34.830268 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:34.830720 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:34.830340 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:35.855651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:35.855610 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:35.856045 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:35.855687 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:36.351907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:36.351834 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:36.830364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:36.830330 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:36.830573 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:36.830402 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:36.935096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:36.935066 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:37.830714 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:37.830678 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body= Nov 02 23:55:37.830907 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:37.830735 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" Nov 02 23:55:38.858119 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:38.858018 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:38.859998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:38.858971 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:38.859998 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:38.859857 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 02 23:55:41.803234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:41.803166 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:42.136918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.136829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:42.678286 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.678252 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:55:42.679001 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.678988 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d664b1cd-f0a0-43a4-b294-ab1f7cde0ef6" containerName="istio-proxy" Nov 02 23:55:42.682985 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.682964 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.759623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.759592 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a1de1-openshift-default-dockercfg-dwkmd\"" Nov 02 23:55:42.818194 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.797252 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:42.929995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.929890 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935563 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935612 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935656 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935841 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935878 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.935912 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.931948 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.936269 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:42.939867 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:42.936306 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.037835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.037798 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.037835 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.037842 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.037862 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.037958 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.037976 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038009 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038034 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038052 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038148 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038356 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038479 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038450 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038658 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038478 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038715 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038694 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.038763 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.038710 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.040774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.040750 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.040929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.040910 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.100465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.100431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.100623 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.100474 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\") pod \"e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.208156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.207985 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:43.359683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.359644 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:43.786989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.786963 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:55:43.787414 ip-10-0-135-179 kubenswrapper[4494]: W1102 23:55:43.787388 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c8a337_7227_4f79_a772_79f619b9e810.slice/crio-3808ac1d435b1a7c33aed1cc58c29c48bba313051e480fbddda7114849348f26 WatchSource:0}: Error finding container 3808ac1d435b1a7c33aed1cc58c29c48bba313051e480fbddda7114849348f26: Status 404 returned error can't find the container with id 3808ac1d435b1a7c33aed1cc58c29c48bba313051e480fbddda7114849348f26 Nov 02 23:55:43.790340 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.790308 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:43.790427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.790380 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:43.790427 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.790418 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:55:43.967533 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.967493 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerStarted","Data":"bc7d08f4f6665790d64c9d761e153314f4398b72693fdacac3052cfc7c33ac23"} Nov 02 23:55:43.967920 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:43.967560 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerStarted","Data":"3808ac1d435b1a7c33aed1cc58c29c48bba313051e480fbddda7114849348f26"} Nov 02 23:55:44.094854 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:44.094744 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podStartSLOduration=2.094722717 podStartE2EDuration="2.094722717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:44.089150728 +0000 UTC m=+4058.519020590" watchObservedRunningTime="2026-11-02 23:55:44.094722717 +0000 UTC m=+4058.524592579" Nov 02 23:55:44.360135 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:44.360060 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:44.365993 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:44.365969 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:44.974774 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:44.974744 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:44.975851 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:44.975836 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:55:46.785557 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:46.785522 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:46.943186 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:46.943142 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:47.175632 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:47.175590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:47.336845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:47.336788 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:47.483560 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:47.483476 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:47.560541 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:55:47.560509 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:55:49.001845 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:49.001811 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:55:52.336221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:52.336160 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:52.486875 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:52.486809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:52.640182 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:52.638605 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:57.550057 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:57.550017 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:55:58.070144 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:55:58.070107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:02.509202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:02.508801 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:07.288688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:07.288655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:12.121748 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:12.121714 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:12.270385 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:12.270353 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:12.416806 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:12.416775 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:17.378030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:17.377987 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:17.594215 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:56:17.594100 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:56:18.773542 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:18.773499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:22.352390 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:22.352350 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:23.335025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.334940 4494 generic.go:356] "Generic (PLEG): container finished" podID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerID="d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499" exitCode=0 Nov 02 23:56:23.335025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.334999 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerDied","Data":"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499"} Nov 02 23:56:23.335498 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.335481 4494 scope.go:122] "RemoveContainer" containerID="d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499" Nov 02 23:56:23.337222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.337164 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:23.337342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.337261 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:23.337342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:23.337293 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:24.361746 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:24.361708 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerStarted","Data":"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7"} Nov 02 23:56:24.496396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:24.496359 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:56:24.497759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:24.497732 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:24.497891 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:24.497783 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:25.496536 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:25.496497 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:25.496925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:25.496561 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:26.496281 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:26.496239 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:26.496500 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:26.496303 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:27.248197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:27.247317 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:56:27.297444 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:27.297409 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:27.441613 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:27.441567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:27.496543 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:27.496512 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:27.496734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:27.496571 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:28.497097 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:28.497058 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:28.497634 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:28.497117 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:29.496885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:29.496848 4494 patch_prober.go:29] interesting pod/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Nov 02 23:56:29.497072 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:29.496905 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Nov 02 23:56:30.501065 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:30.501034 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:56:30.502111 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:30.502088 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:56:32.228298 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:32.228259 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:36.812260 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.812228 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 02 23:56:36.815551 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.815533 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:36.870310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.870284 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5e7e9e-serving-cert\"" Nov 02 23:56:36.924370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.924342 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 02 23:56:36.975041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.975010 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:36.975333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.975304 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:36.975442 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.975420 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:36.975491 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:36.975482 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rm9\" (UniqueName: \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.007564 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.007491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:37.085745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.083556 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.085745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.083891 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.085745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.084032 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.085745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.084165 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rm9\" (UniqueName: \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.093752 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.093640 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.107766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.106854 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.107766 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.107458 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.153744 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.153717 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rm9\" (UniqueName: \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\") pod \"maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.165688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.165632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:37.307745 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.307714 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:37.427446 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.427414 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:37.807919 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.807832 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 02 23:56:37.819525 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:37.819500 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:56:38.486802 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:38.486770 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" event={"ID":"1c89e234-56b7-432c-9aa3-c4c7b214a7f3","Type":"ContainerStarted","Data":"0cbed8ad0aa448ab942005793dffb0d200c8f02c50661b3b2eb311765453f5a7"} Nov 02 23:56:39.495008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:39.494973 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" event={"ID":"1c89e234-56b7-432c-9aa3-c4c7b214a7f3","Type":"ContainerStarted","Data":"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9"} Nov 02 23:56:39.495457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:39.495044 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:39.614115 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:39.614049 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" podStartSLOduration=2.780711926 podStartE2EDuration="3.614035565s" totalImagesPullingTime="833.323639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:56:39.613625728 +0000 UTC m=+4114.043495588" watchObservedRunningTime="2026-11-02 23:56:39.614035565 +0000 UTC m=+4114.043905431" Nov 02 23:56:42.164291 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:42.164251 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:42.260924 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:42.260892 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:56:43.318127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:43.318096 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:45.510200 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:45.510139 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 02 23:56:47.108029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:47.107992 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:47.592372 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:56:47.592327 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:56:51.884687 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:51.884650 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:52.042796 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.042759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:52.189989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.189953 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:52.624303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.624220 4494 generic.go:356] "Generic (PLEG): container finished" podID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerID="b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3" exitCode=0 Nov 02 23:56:52.624303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.624290 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerDied","Data":"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3"} Nov 02 23:56:52.624721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.624706 4494 scope.go:122] "RemoveContainer" containerID="b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3" Nov 02 23:56:52.626489 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.626463 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:52.626580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.626533 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:52.626580 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:52.626564 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:56:53.636228 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:53.636148 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerStarted","Data":"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44"} Nov 02 23:56:54.296299 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:54.296260 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:56:54.296579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:54.296515 4494 patch_prober.go:29] interesting pod/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Nov 02 23:56:54.296629 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:54.296570 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Nov 02 23:56:55.296398 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:55.296352 4494 patch_prober.go:29] interesting pod/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Nov 02 23:56:55.296820 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:55.296419 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Nov 02 23:56:56.296486 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:56.296454 4494 patch_prober.go:29] interesting pod/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Nov 02 23:56:56.296894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:56.296504 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Nov 02 23:56:57.087341 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:57.087291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:56:57.296941 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:57.296904 4494 patch_prober.go:29] interesting pod/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Nov 02 23:56:57.297402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:57.296961 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Nov 02 23:56:58.296885 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:58.296853 4494 patch_prober.go:29] interesting pod/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body= Nov 02 23:56:58.297130 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:58.297107 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" Nov 02 23:56:59.301402 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:59.301365 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:56:59.301803 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:59.301706 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:56:59.302414 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:56:59.302394 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 02 23:57:02.081930 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:02.081843 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:07.021792 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:07.021757 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:11.912051 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:11.912018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:16.772756 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:16.772722 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:17.561249 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:57:17.561213 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:57:19.117841 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:19.117783 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:21.523407 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.523361 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:57:21.540233 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.540199 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:57:21.540562 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.540540 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.552777 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.552752 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.552909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.552784 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhs5w\" (UniqueName: \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.552909 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.552907 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.553033 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.552977 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.604583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.604553 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a1de1-serving-cert\"" Nov 02 23:57:21.605535 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.605487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:21.654142 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.654109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.654372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.654240 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.654372 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.654359 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.654514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.654385 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhs5w\" (UniqueName: \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.654755 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:57:21.654726 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-serving-cert: secret "maas-api-e2e-inf-a-3a1de1-serving-cert" not found Nov 02 23:57:21.654884 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:57:21.654834 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls podName:0088a37e-cb30-4234-a595-422447f6b821 nodeName:}" failed. No retries permitted until 2026-11-02 23:57:22.154813913 +0000 UTC m=+4156.584683768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls") pod "maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" (UID: "0088a37e-cb30-4234-a595-422447f6b821") : secret "maas-api-e2e-inf-a-3a1de1-serving-cert" not found Nov 02 23:57:21.655155 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.655129 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.657226 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.657166 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.734691 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.729722 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhs5w\" (UniqueName: \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:21.768093 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.768060 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:57:21.792478 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.792372 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:21.937624 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:21.937589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:22.168387 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:22.168355 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:22.170934 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:22.170911 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:22.459465 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:22.459367 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:22.889511 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:22.889477 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:57:22.952242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:22.952209 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" event={"ID":"0088a37e-cb30-4234-a595-422447f6b821","Type":"ContainerStarted","Data":"877e8f893673b9dd6a73d95e5efc20248fdf4cad8feadcc403eab57a3e11ddfa"} Nov 02 23:57:23.966595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:23.966493 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" event={"ID":"0088a37e-cb30-4234-a595-422447f6b821","Type":"ContainerStarted","Data":"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913"} Nov 02 23:57:23.966595 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:23.966568 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:24.086497 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:24.086442 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" podStartSLOduration=2.297543448 podStartE2EDuration="3.086425978s" totalImagesPullingTime="788.88253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:24.08435484 +0000 UTC m=+4158.514224703" watchObservedRunningTime="2026-11-02 23:57:24.086425978 +0000 UTC m=+4158.516295839" Nov 02 23:57:26.878868 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:26.878828 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:29.981294 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:29.981266 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:57:31.960457 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:31.960409 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:32.106333 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:32.106298 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:32.773240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:32.773141 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:32.773467 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:32.773220 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:33.048024 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.047933 4494 generic.go:356] "Generic (PLEG): container finished" podID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerID="11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4" exitCode=0 Nov 02 23:57:33.048450 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.048017 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerDied","Data":"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4"} Nov 02 23:57:33.048523 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.048471 4494 scope.go:122] "RemoveContainer" containerID="11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4" Nov 02 23:57:33.050310 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.050286 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:33.050396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.050354 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:33.050396 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:33.050385 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:34.061569 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:34.061534 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerStarted","Data":"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6"} Nov 02 23:57:34.854198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:34.853584 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:57:34.854198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:34.853709 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:34.854198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:34.853765 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:35.853737 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:35.853695 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:35.854309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:35.853761 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:36.847663 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:36.847621 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:36.847972 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:36.847938 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:37.282765 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:37.282730 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:37.687334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:37.687302 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:37.847464 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:37.847431 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:37.848064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:37.847950 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:37.848278 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:37.848252 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:38.200222 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:38.200184 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:38.852196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:38.851621 4494 patch_prober.go:29] interesting pod/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Nov 02 23:57:38.852196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:38.851686 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Nov 02 23:57:39.846473 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:39.846441 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:57:39.846936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:39.846922 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:57:39.847662 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:39.847646 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:57:42.702095 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:42.702060 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:44.055199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:44.053697 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:47.594100 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:57:47.594019 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:57:48.256644 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:48.256608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:52.250686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:52.250632 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:53.222818 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.222781 4494 generic.go:356] "Generic (PLEG): container finished" podID="49c8a337-7227-4f79-a772-79f619b9e810" containerID="bc7d08f4f6665790d64c9d761e153314f4398b72693fdacac3052cfc7c33ac23" exitCode=0 Nov 02 23:57:53.223066 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.222857 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerDied","Data":"bc7d08f4f6665790d64c9d761e153314f4398b72693fdacac3052cfc7c33ac23"} Nov 02 23:57:53.223342 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.223323 4494 scope.go:122] "RemoveContainer" containerID="bc7d08f4f6665790d64c9d761e153314f4398b72693fdacac3052cfc7c33ac23" Nov 02 23:57:53.225127 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.225098 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:53.225221 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.225207 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:53.225273 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.225247 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 02 23:57:53.329140 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:53.329103 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:54.256635 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:54.256599 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerStarted","Data":"a10566fca513d0923e624f387c95eae73cec5b6d7520a47a439fa94605ea5855"} Nov 02 23:57:54.360614 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:54.360573 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:57:54.361071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:54.360781 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:54.361071 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:54.360834 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:55.360689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:55.360650 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:55.361329 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:55.361272 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:56.360297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:56.360259 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:56.360495 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:56.360318 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:57.360855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:57.360818 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:57.361297 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:57.360872 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:58.360734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:58.360699 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:58.361076 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:58.360758 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:58.479596 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:58.479564 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:58.656933 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:58.656850 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:59.320445 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:59.320378 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:59.360156 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:59.360126 4494 patch_prober.go:29] interesting pod/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Nov 02 23:57:59.360378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:59.360191 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Nov 02 23:57:59.472334 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:59.472289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:57:59.975433 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:57:59.975395 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:58:00.517383 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:00.517352 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:58:00.519897 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:00.519591 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:58:01.989151 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:01.989119 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:58:03.370187 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:03.370137 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:04.393438 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:04.393387 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:05.842153 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:05.842103 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:07.761487 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:07.761447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:08.214197 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:08.214107 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:11.767121 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:11.767083 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:12.539707 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:12.538653 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:58:12.783967 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:12.783933 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:12.929989 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:12.929953 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:13.235105 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:13.235008 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:17.534008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.533940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:17.539303 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.539267 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" podUID="b746fac7-2995-41ea-b86a-75537acf4162" containerName="payload-pre-processing" containerID="cri-o://06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c" gracePeriod=25 Nov 02 23:58:17.568849 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:58:17.568810 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:58:17.803741 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.803678 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:58:17.875116 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.875086 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\") pod \"b746fac7-2995-41ea-b86a-75537acf4162\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " Nov 02 23:58:17.875328 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.875256 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\" (UniqueName: \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\") pod \"b746fac7-2995-41ea-b86a-75537acf4162\" (UID: \"b746fac7-2995-41ea-b86a-75537acf4162\") " Nov 02 23:58:17.875548 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.875527 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume" pod "b746fac7-2995-41ea-b86a-75537acf4162" (UID: "b746fac7-2995-41ea-b86a-75537acf4162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:17.875837 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.875812 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b746fac7-2995-41ea-b86a-75537acf4162-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:17.877556 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.877536 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b" pod "b746fac7-2995-41ea-b86a-75537acf4162" (UID: "b746fac7-2995-41ea-b86a-75537acf4162"). InnerVolumeSpecName "kube-api-access-ph44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:17.977025 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:17.976990 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph44b\" (UniqueName: \"kubernetes.io/projected/b746fac7-2995-41ea-b86a-75537acf4162-kube-api-access-ph44b\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:18.458494 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.458463 4494 generic.go:356] "Generic (PLEG): container finished" podID="b746fac7-2995-41ea-b86a-75537acf4162" containerID="06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c" exitCode=0 Nov 02 23:58:18.458685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.458508 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" Nov 02 23:58:18.458685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.458545 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" event={"ID":"b746fac7-2995-41ea-b86a-75537acf4162","Type":"ContainerDied","Data":"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c"} Nov 02 23:58:18.458685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.458602 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q" event={"ID":"b746fac7-2995-41ea-b86a-75537acf4162","Type":"ContainerDied","Data":"8093de9cd18ed121b93af81f90c0d466778d344cfdd5a6a1295a122f28bb0726"} Nov 02 23:58:18.458685 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.458630 4494 scope.go:122] "RemoveContainer" containerID="06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c" Nov 02 23:58:18.469258 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.469238 4494 scope.go:122] "RemoveContainer" containerID="06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c" Nov 02 23:58:18.469539 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:58:18.469520 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c\": container with ID starting with 06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c not found: ID does not exist" containerID="06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c" Nov 02 23:58:18.469583 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.469547 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c"} err="failed to get container status \"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c\": rpc error: code = NotFound desc = could not find container \"06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c\": container with ID starting with 06f95d39326a9cd6e41671fc6c91d5e31962143da4ec77c630e5d5e99b77489c not found: ID does not exist" Nov 02 23:58:18.681011 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.680976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:18.685160 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.685130 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:58:18.689721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:18.689694 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f7002-bbb789fdc-p9z7q"] Nov 02 23:58:19.462106 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:19.462075 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:19.694719 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:19.694684 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b746fac7-2995-41ea-b86a-75537acf4162" path="/var/lib/kubelet/pods/b746fac7-2995-41ea-b86a-75537acf4162/volumes" Nov 02 23:58:22.668006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:22.667967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:23.932454 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:23.932415 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:28.656660 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:28.656624 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:29.344929 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:29.344881 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:30.526827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:30.526793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:30.674959 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:30.674926 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:33.772037 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:33.771995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:34.579926 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:34.579894 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:35.556652 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:35.556616 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:40.441282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:40.441248 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:40.588659 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:40.588620 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:40.735960 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:40.735875 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:42.108549 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:42.108518 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:42.715472 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:42.715443 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:58:42.715690 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:42.715670 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" containerID="cri-o://ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6" gracePeriod=30 Nov 02 23:58:42.736096 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:42.736054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:45.219965 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:45.219914 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:45.364612 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:45.364577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:45.516198 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:45.516081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:45.772236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:45.772122 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:45.916936 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:45.916889 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:47.564052 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:58:47.564010 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:58:48.002616 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.002588 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:58:48.062395 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062334 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062412 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062537 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062599 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062572 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062668 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062743 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062776 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.062809 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062782 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:58:48.063030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062828 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.063030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062859 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:58:48.063030 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062925 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\") pod \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\" (UID: \"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa\") " Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.062922 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.063551 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.063584 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.063600 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.063552 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:58:48.063654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.063593 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:58:48.065654 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.065615 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:58:48.065838 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.065812 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "kube-api-access-lmrdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:48.066064 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.066043 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:48.066202 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.066150 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy" pod "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" (UID: "b64ead7f-e1af-4e13-ac03-faef9ddf6cfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:58:48.165006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.164976 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.165006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.165002 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.165006 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.165012 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.165284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.165022 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmrdg\" (UniqueName: \"kubernetes.io/projected/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-kube-api-access-lmrdg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.165284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.165034 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.165284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.165042 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:58:48.804815 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.804778 4494 generic.go:356] "Generic (PLEG): container finished" podID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerID="ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6" exitCode=0 Nov 02 23:58:48.805309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.804827 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" Nov 02 23:58:48.805309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.804863 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerDied","Data":"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6"} Nov 02 23:58:48.805309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.804933 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5" event={"ID":"b64ead7f-e1af-4e13-ac03-faef9ddf6cfa","Type":"ContainerDied","Data":"b6304cd50245768993e282e400815b9c3fc7c352c08f38a07da47cd7c5ff7682"} Nov 02 23:58:48.805309 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.804965 4494 scope.go:122] "RemoveContainer" containerID="ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6" Nov 02 23:58:48.816370 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.816350 4494 scope.go:122] "RemoveContainer" containerID="11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4" Nov 02 23:58:48.827295 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.827264 4494 scope.go:122] "RemoveContainer" containerID="ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6" Nov 02 23:58:48.827555 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:58:48.827536 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6\": container with ID starting with ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6 not found: ID does not exist" containerID="ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6" Nov 02 23:58:48.827638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.827564 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6"} err="failed to get container status \"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6\": rpc error: code = NotFound desc = could not find container \"ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6\": container with ID starting with ffb67250362e851329128545f5d658572f7243e29adeae29f8cd5844067b60a6 not found: ID does not exist" Nov 02 23:58:48.827638 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.827583 4494 scope.go:122] "RemoveContainer" containerID="11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4" Nov 02 23:58:48.827861 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:58:48.827838 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4\": container with ID starting with 11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4 not found: ID does not exist" containerID="11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4" Nov 02 23:58:48.827935 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:48.827862 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4"} err="failed to get container status \"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4\": rpc error: code = NotFound desc = could not find container \"11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4\": container with ID starting with 11b10522f2a492e95989248c37e500e7d79632ac949f9b7d04eeb7403af695d4 not found: ID does not exist" Nov 02 23:58:49.022979 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:49.022934 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:58:49.073597 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:49.073512 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7097dd-openshift-default-5c45f6d47c-htdm5"] Nov 02 23:58:49.696546 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:49.696511 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" path="/var/lib/kubelet/pods/b64ead7f-e1af-4e13-ac03-faef9ddf6cfa/volumes" Nov 02 23:58:50.789758 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:50.789718 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:51.724689 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:51.724640 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:52.717248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:52.717199 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:52.861423 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:52.861387 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:55.666677 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:55.666644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:57.446364 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:57.446331 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:59.848731 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:59.848698 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:58:59.849196 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:59.848921 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" podUID="0088a37e-cb30-4234-a595-422447f6b821" containerName="maas-api" containerID="cri-o://4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913" gracePeriod=30 Nov 02 23:58:59.950819 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:59.950778 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:58:59.976420 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:58:59.976383 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" podUID="0088a37e-cb30-4234-a595-422447f6b821" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.31:8443/health\": dial tcp 10.133.1.31:8443: connect: connection refused" Nov 02 23:59:00.185022 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.184999 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:59:00.302236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.300704 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\") pod \"0088a37e-cb30-4234-a595-422447f6b821\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " Nov 02 23:59:00.302236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.300816 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\") pod \"0088a37e-cb30-4234-a595-422447f6b821\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " Nov 02 23:59:00.302236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.300874 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\" (UniqueName: \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\") pod \"0088a37e-cb30-4234-a595-422447f6b821\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " Nov 02 23:59:00.302236 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.300981 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") pod \"0088a37e-cb30-4234-a595-422447f6b821\" (UID: \"0088a37e-cb30-4234-a595-422447f6b821\") " Nov 02 23:59:00.303981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.303707 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca" pod "0088a37e-cb30-4234-a595-422447f6b821" (UID: "0088a37e-cb30-4234-a595-422447f6b821"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:00.310918 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.310871 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 02 23:59:00.319759 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.319217 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs" pod "0088a37e-cb30-4234-a595-422447f6b821" (UID: "0088a37e-cb30-4234-a595-422447f6b821"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:00.319988 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.319955 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls" pod "0088a37e-cb30-4234-a595-422447f6b821" (UID: "0088a37e-cb30-4234-a595-422447f6b821"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:00.320143 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.320108 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w" pod "0088a37e-cb30-4234-a595-422447f6b821" (UID: "0088a37e-cb30-4234-a595-422447f6b821"). InnerVolumeSpecName "kube-api-access-xhs5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:00.402999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.402959 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:00.402999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.402985 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0088a37e-cb30-4234-a595-422447f6b821-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:00.402999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.402996 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0088a37e-cb30-4234-a595-422447f6b821-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:00.402999 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.403006 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhs5w\" (UniqueName: \"kubernetes.io/projected/0088a37e-cb30-4234-a595-422447f6b821-kube-api-access-xhs5w\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:00.921374 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.921342 4494 generic.go:356] "Generic (PLEG): container finished" podID="0088a37e-cb30-4234-a595-422447f6b821" containerID="4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913" exitCode=0 Nov 02 23:59:00.921894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.921391 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" Nov 02 23:59:00.921894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.921421 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" event={"ID":"0088a37e-cb30-4234-a595-422447f6b821","Type":"ContainerDied","Data":"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913"} Nov 02 23:59:00.921894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.921475 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj" event={"ID":"0088a37e-cb30-4234-a595-422447f6b821","Type":"ContainerDied","Data":"877e8f893673b9dd6a73d95e5efc20248fdf4cad8feadcc403eab57a3e11ddfa"} Nov 02 23:59:00.921894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.921498 4494 scope.go:122] "RemoveContainer" containerID="4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913" Nov 02 23:59:00.935242 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.935216 4494 scope.go:122] "RemoveContainer" containerID="4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913" Nov 02 23:59:00.935606 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:59:00.935586 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913\": container with ID starting with 4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913 not found: ID does not exist" containerID="4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913" Nov 02 23:59:00.935683 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:00.935614 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913"} err="failed to get container status \"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913\": rpc error: code = NotFound desc = could not find container \"4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913\": container with ID starting with 4fd9559b543a87aa1ebf702e62925aee9523983e227624918deb84fed3efb913 not found: ID does not exist" Nov 02 23:59:01.101259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:01.096351 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:59:01.101259 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:01.100129 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a1de1-69cc6d5787-g9ldj"] Nov 02 23:59:01.116568 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:01.116536 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:01.694995 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:01.694962 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0088a37e-cb30-4234-a595-422447f6b821" path="/var/lib/kubelet/pods/0088a37e-cb30-4234-a595-422447f6b821/volumes" Nov 02 23:59:03.018520 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:03.018433 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:03.841099 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:03.841063 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:04.002734 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:04.002696 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:04.422413 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:04.422383 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:04.426981 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:04.426952 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:59:04.428949 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:04.428921 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" containerID="cri-o://4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7" gracePeriod=30 Nov 02 23:59:05.446811 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:05.446760 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:05.932326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:05.932289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:05.933827 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:05.933799 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:59:05.934108 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:05.934067 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" containerID="cri-o://a10566fca513d0923e624f387c95eae73cec5b6d7520a47a439fa94605ea5855" gracePeriod=30 Nov 02 23:59:06.752219 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:06.752140 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:06.901717 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:06.901675 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:08.382499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:08.382466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:08.535440 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:08.535397 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:08.703773 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:08.703717 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:09.660017 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.659990 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:59:09.749367 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.749327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:09.757284 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757262 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757324 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757419 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757349 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757434 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757499 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757485 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757579 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757539 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757583 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:09.757642 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757617 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757652 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757751 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757686 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\") pod \"61531ff1-458f-420a-92ed-8df67e3eb4bb\" (UID: \"61531ff1-458f-420a-92ed-8df67e3eb4bb\") " Nov 02 23:59:09.757862 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757806 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:09.757946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.757897 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:09.758070 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758046 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:09.758253 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758232 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:09.758411 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758346 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.758688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758437 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.758688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758456 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61531ff1-458f-420a-92ed-8df67e3eb4bb-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.758688 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.758469 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.760029 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.760002 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:09.760326 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.760310 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "kube-api-access-w9vfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:09.760561 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.760541 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:09.760767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.760750 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo" pod "61531ff1-458f-420a-92ed-8df67e3eb4bb" (UID: "61531ff1-458f-420a-92ed-8df67e3eb4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:59:09.860089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.860056 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.860089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.860085 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.860089 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.860095 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.860378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.860107 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61531ff1-458f-420a-92ed-8df67e3eb4bb-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.860378 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.860117 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9vfq\" (UniqueName: \"kubernetes.io/projected/61531ff1-458f-420a-92ed-8df67e3eb4bb-kube-api-access-w9vfq\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:09.901894 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:09.901860 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:10.112767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.112675 4494 generic.go:356] "Generic (PLEG): container finished" podID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerID="4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7" exitCode=0 Nov 02 23:59:10.112767 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.112743 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" Nov 02 23:59:10.113007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.112769 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerDied","Data":"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7"} Nov 02 23:59:10.113007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.112826 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s" event={"ID":"61531ff1-458f-420a-92ed-8df67e3eb4bb","Type":"ContainerDied","Data":"892c130d6f34390d7a1dc3b37c871842a55911816464026a10f5ac93e3cdb74f"} Nov 02 23:59:10.113007 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.112863 4494 scope.go:122] "RemoveContainer" containerID="4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7" Nov 02 23:59:10.125332 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.125318 4494 scope.go:122] "RemoveContainer" containerID="d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499" Nov 02 23:59:10.135343 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.135322 4494 scope.go:122] "RemoveContainer" containerID="4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7" Nov 02 23:59:10.135630 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:59:10.135605 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7\": container with ID starting with 4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7 not found: ID does not exist" containerID="4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7" Nov 02 23:59:10.135686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.135641 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7"} err="failed to get container status \"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7\": rpc error: code = NotFound desc = could not find container \"4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7\": container with ID starting with 4756e6a6b1afe8dc08855e562386b19501a629bbd9a648a68671d07c4f594af7 not found: ID does not exist" Nov 02 23:59:10.135686 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.135660 4494 scope.go:122] "RemoveContainer" containerID="d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499" Nov 02 23:59:10.135913 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:59:10.135892 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499\": container with ID starting with d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499 not found: ID does not exist" containerID="d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499" Nov 02 23:59:10.135976 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.135924 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499"} err="failed to get container status \"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499\": rpc error: code = NotFound desc = could not find container \"d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499\": container with ID starting with d98a4deb0052658c4c00615a8809201cf82838bfffdbe3cf1b3ffca05b904499 not found: ID does not exist" Nov 02 23:59:10.309869 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.309836 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:59:10.359863 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:10.359832 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-18a6d8-openshift-default-6c6c597555-x862s"] Nov 02 23:59:11.126961 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.126922 4494 generic.go:356] "Generic (PLEG): container finished" podID="49c8a337-7227-4f79-a772-79f619b9e810" containerID="a10566fca513d0923e624f387c95eae73cec5b6d7520a47a439fa94605ea5855" exitCode=0 Nov 02 23:59:11.127468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.127106 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerDied","Data":"a10566fca513d0923e624f387c95eae73cec5b6d7520a47a439fa94605ea5855"} Nov 02 23:59:11.127468 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.127204 4494 scope.go:122] "RemoveContainer" containerID="bc7d08f4f6665790d64c9d761e153314f4398b72693fdacac3052cfc7c33ac23" Nov 02 23:59:11.198606 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.198580 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:59:11.280693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280606 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.280693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280647 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.280693 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280677 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280703 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280780 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280844 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280886 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281008 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.280968 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281022 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\") pod \"49c8a337-7227-4f79-a772-79f619b9e810\" (UID: \"49c8a337-7227-4f79-a772-79f619b9e810\") " Nov 02 23:59:11.281321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281057 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:11.281321 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281097 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:11.281669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281345 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:11.281669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281370 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:11.281669 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281380 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:11.281847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281803 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.281847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281822 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.281847 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281835 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c8a337-7227-4f79-a772-79f619b9e810-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.282041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281849 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.282041 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.281865 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.283262 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.283238 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:59:11.283408 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.283265 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "kube-api-access-bhsvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:11.283630 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.283615 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:59:11.283721 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.283697 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token" pod "49c8a337-7227-4f79-a772-79f619b9e810" (UID: "49c8a337-7227-4f79-a772-79f619b9e810"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:11.383565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.383530 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c8a337-7227-4f79-a772-79f619b9e810-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.383565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.383559 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c8a337-7227-4f79-a772-79f619b9e810-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.383565 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.383569 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsvm\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-kube-api-access-bhsvm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.383824 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.383579 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c8a337-7227-4f79-a772-79f619b9e810-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 23:59:11.695132 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:11.695097 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" path="/var/lib/kubelet/pods/61531ff1-458f-420a-92ed-8df67e3eb4bb/volumes" Nov 02 23:59:12.138282 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:12.138165 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" Nov 02 23:59:12.138674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:12.138198 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p" event={"ID":"49c8a337-7227-4f79-a772-79f619b9e810","Type":"ContainerDied","Data":"3808ac1d435b1a7c33aed1cc58c29c48bba313051e480fbddda7114849348f26"} Nov 02 23:59:12.138674 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:12.138345 4494 scope.go:122] "RemoveContainer" containerID="a10566fca513d0923e624f387c95eae73cec5b6d7520a47a439fa94605ea5855" Nov 02 23:59:12.357389 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:12.357358 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:59:12.407984 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:12.407903 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a1de1-openshift-default-6cf4db49df-rrm8p"] Nov 02 23:59:13.694269 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:13.694234 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c8a337-7227-4f79-a772-79f619b9e810" path="/var/lib/kubelet/pods/49c8a337-7227-4f79-a772-79f619b9e810/volumes" Nov 02 23:59:13.840541 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:13.840504 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:17.562601 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:59:17.562501 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:59:18.735224 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:18.735156 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:19.808726 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:19.808686 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:22.677701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:22.677666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:23.249855 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:23.249817 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:23.707227 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:23.707191 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:28.056325 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:28.056292 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:29.211514 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:29.211478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:29.605701 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:29.605618 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:29.839974 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:29.839941 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:32.730592 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:32.730563 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:34.683729 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:34.683698 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:35.724925 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:35.724885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:35.870917 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:35.870880 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:37.354946 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:37.354906 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:37.898651 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:37.898607 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:38.709040 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:38.709000 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:39.916369 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:39.916332 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:44.756322 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:44.756281 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:45.100240 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:45.100115 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:45.688199 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:45.684667 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:45.870248 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:45.870128 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:47.561978 ip-10-0-135-179 kubenswrapper[4494]: E1102 23:59:47.561929 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 02 23:59:50.453234 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:50.453161 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:50.595185 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:50.595136 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:55.352455 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:55.352420 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 02 23:59:59.751141 ip-10-0-135-179 kubenswrapper[4494]: I1102 23:59:59.751097 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:00.183258 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.183217 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:00.184060 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184038 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b746fac7-2995-41ea-b86a-75537acf4162" containerName="payload-pre-processing" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184064 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b746fac7-2995-41ea-b86a-75537acf4162" containerName="payload-pre-processing" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184087 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184094 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184135 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0088a37e-cb30-4234-a595-422447f6b821" containerName="maas-api" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184143 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0088a37e-cb30-4234-a595-422447f6b821" containerName="maas-api" Nov 03 00:00:00.184153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184153 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184164 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184201 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184209 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184223 4494 cpu_manager.go:414] "containerMap: removing container" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184230 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184242 4494 cpu_manager.go:414] "containerMap: removing container" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184247 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184362 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b746fac7-2995-41ea-b86a-75537acf4162" containerName="payload-pre-processing" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184373 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184382 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0088a37e-cb30-4234-a595-422447f6b821" containerName="maas-api" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184390 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184397 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64ead7f-e1af-4e13-ac03-faef9ddf6cfa" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184404 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.184445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.184409 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c8a337-7227-4f79-a772-79f619b9e810" containerName="istio-proxy" Nov 03 00:00:00.188153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.188137 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.199238 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.199209 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:00.199818 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.199805 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.199864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.199820 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.199937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.199925 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61531ff1-458f-420a-92ed-8df67e3eb4bb" containerName="istio-proxy" Nov 03 00:00:00.203488 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.203472 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.234102 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.234075 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:00.238516 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.238492 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:00.238665 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.238638 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.242709 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.242691 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:00.242855 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.242813 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.246689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.246669 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.260512 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.260490 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:00.288050 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.288020 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:00.334942 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.334911 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.335275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.335120 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdnv\" (UniqueName: \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.350106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.350069 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.371278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.350437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9x9\" (UniqueName: \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.398326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.398283 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5nhs\" (UniqueName: \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.402813 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.398585 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qml\" (UniqueName: \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.403118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.403093 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.403419 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.403383 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cl48\" (UniqueName: \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.403738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.403716 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.403922 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.403871 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.404748 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.404492 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:00:00.406713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.406680 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:00.416427 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.412836 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:00.418678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.417410 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:00.444345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.443208 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:00.507413 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507382 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.507613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507436 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.507613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507489 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.507613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507600 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdnv\" (UniqueName: \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.507799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507627 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.507799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507712 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9x9\" (UniqueName: \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.507799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507749 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5nhs\" (UniqueName: \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.507799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507775 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qml\" (UniqueName: \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.508016 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507815 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.508016 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.507881 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cl48\" (UniqueName: \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.509216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.508227 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.509216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.508787 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.509216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.508919 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.509216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.509141 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.509538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.509280 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.571095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.570719 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5nhs\" (UniqueName: \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\") pod \"maas-api-key-cleanup-29894400-pq625\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.571095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.571032 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qml\" (UniqueName: \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\") pod \"maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.572431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.571878 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cl48\" (UniqueName: \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\") pod \"maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.572431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.572389 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9x9\" (UniqueName: \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.575770 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.575558 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:00.575964 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.575948 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdnv\" (UniqueName: \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\") pod \"maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.820891 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.820804 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:00.829366 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.829346 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:00.854591 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.854560 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:00.864266 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.864235 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:00.966530 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:00.966255 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:00.976034 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:00:00.976007 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba8d1b9_94bd_4885_9f4a_ea24952cb1b8.slice/crio-ed2178d2c782ae20512ae6a104baa1bc47bab4f480f5f6a029ff9412143b69aa WatchSource:0}: Error finding container ed2178d2c782ae20512ae6a104baa1bc47bab4f480f5f6a029ff9412143b69aa: Status 404 returned error can't find the container with id ed2178d2c782ae20512ae6a104baa1bc47bab4f480f5f6a029ff9412143b69aa Nov 03 00:00:01.218600 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.218575 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:01.218970 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:00:01.218948 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f4f435_8294_4190_840e_43e1da0caf9e.slice/crio-57cda572de0168c37e38eb40fe3ef83f12f2d34b201a05d6c2926edad54ae5f7 WatchSource:0}: Error finding container 57cda572de0168c37e38eb40fe3ef83f12f2d34b201a05d6c2926edad54ae5f7: Status 404 returned error can't find the container with id 57cda572de0168c37e38eb40fe3ef83f12f2d34b201a05d6c2926edad54ae5f7 Nov 03 00:00:01.237067 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.237035 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:01.239807 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:00:01.239775 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5e7357_60d7_407b_89c1_6f4b1ee64dc7.slice/crio-7378a104ae3b7ebf7e9211e1bcc0055bb53398d49b91bca7f0b79906ddcaef55 WatchSource:0}: Error finding container 7378a104ae3b7ebf7e9211e1bcc0055bb53398d49b91bca7f0b79906ddcaef55: Status 404 returned error can't find the container with id 7378a104ae3b7ebf7e9211e1bcc0055bb53398d49b91bca7f0b79906ddcaef55 Nov 03 00:00:01.267794 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.267770 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:01.269037 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:00:01.269008 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499e1125_ce2a_4007_bcf5_6c6ceaf37ee7.slice/crio-1eb3f9aad3d4e15acf30499ed19055cfce01ca87d165dc37baa192b8f163bdc7 WatchSource:0}: Error finding container 1eb3f9aad3d4e15acf30499ed19055cfce01ca87d165dc37baa192b8f163bdc7: Status 404 returned error can't find the container with id 1eb3f9aad3d4e15acf30499ed19055cfce01ca87d165dc37baa192b8f163bdc7 Nov 03 00:00:01.308266 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.308237 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:01.309325 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:00:01.309299 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9007ede0_103c_4ea4_9933_43d4a5b74583.slice/crio-6b2e5b019ebfdf0a0a5663d38c1d70b864b31a2ff814f4a9271ba0e378edef95 WatchSource:0}: Error finding container 6b2e5b019ebfdf0a0a5663d38c1d70b864b31a2ff814f4a9271ba0e378edef95: Status 404 returned error can't find the container with id 6b2e5b019ebfdf0a0a5663d38c1d70b864b31a2ff814f4a9271ba0e378edef95 Nov 03 00:00:01.501890 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.501854 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerStarted","Data":"1eb3f9aad3d4e15acf30499ed19055cfce01ca87d165dc37baa192b8f163bdc7"} Nov 03 00:00:01.503337 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.503311 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerStarted","Data":"cba30fd36bbca136fec0e4d5d4f6e18215951e85fa740fbcb0c73c19d0ee83a6"} Nov 03 00:00:01.503483 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.503369 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerStarted","Data":"ed2178d2c782ae20512ae6a104baa1bc47bab4f480f5f6a029ff9412143b69aa"} Nov 03 00:00:01.504587 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.504565 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerStarted","Data":"57cda572de0168c37e38eb40fe3ef83f12f2d34b201a05d6c2926edad54ae5f7"} Nov 03 00:00:01.505688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.505664 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerStarted","Data":"7378a104ae3b7ebf7e9211e1bcc0055bb53398d49b91bca7f0b79906ddcaef55"} Nov 03 00:00:01.506803 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.506778 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerStarted","Data":"6b2e5b019ebfdf0a0a5663d38c1d70b864b31a2ff814f4a9271ba0e378edef95"} Nov 03 00:00:01.622521 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.622464 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" podStartSLOduration=1.622449868 podStartE2EDuration="1.622449868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:01.619955902 +0000 UTC m=+4316.049825763" watchObservedRunningTime="2026-11-03 00:00:01.622449868 +0000 UTC m=+4316.052319775" Nov 03 00:00:01.629900 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:01.629874 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:02.516136 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.516101 4494 generic.go:356] "Generic (PLEG): container finished" podID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerID="86d9e84bbca8bcc3d7384e3b324ad2fd1a116e50a178a2645d62fe31f2f87850" exitCode=60 Nov 03 00:00:02.516611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.516208 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerDied","Data":"86d9e84bbca8bcc3d7384e3b324ad2fd1a116e50a178a2645d62fe31f2f87850"} Nov 03 00:00:02.516611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.516507 4494 scope.go:122] "RemoveContainer" containerID="86d9e84bbca8bcc3d7384e3b324ad2fd1a116e50a178a2645d62fe31f2f87850" Nov 03 00:00:02.517963 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.517940 4494 generic.go:356] "Generic (PLEG): container finished" podID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerID="cba30fd36bbca136fec0e4d5d4f6e18215951e85fa740fbcb0c73c19d0ee83a6" exitCode=60 Nov 03 00:00:02.518085 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.518009 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerDied","Data":"cba30fd36bbca136fec0e4d5d4f6e18215951e85fa740fbcb0c73c19d0ee83a6"} Nov 03 00:00:02.518367 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.518335 4494 scope.go:122] "RemoveContainer" containerID="cba30fd36bbca136fec0e4d5d4f6e18215951e85fa740fbcb0c73c19d0ee83a6" Nov 03 00:00:02.519739 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.519712 4494 generic.go:356] "Generic (PLEG): container finished" podID="58f4f435-8294-4190-840e-43e1da0caf9e" containerID="646750914a0f84e2136a6eef586ad25af05f1a24baf8b96ebc7014e5a3cf5d0a" exitCode=60 Nov 03 00:00:02.519944 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.519743 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerDied","Data":"646750914a0f84e2136a6eef586ad25af05f1a24baf8b96ebc7014e5a3cf5d0a"} Nov 03 00:00:02.520210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.520025 4494 scope.go:122] "RemoveContainer" containerID="646750914a0f84e2136a6eef586ad25af05f1a24baf8b96ebc7014e5a3cf5d0a" Nov 03 00:00:02.521610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.521586 4494 generic.go:356] "Generic (PLEG): container finished" podID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerID="621588198b060c8f74ce27e7b43df505fd49bd845ce67f529baab8a9ea0d4fad" exitCode=60 Nov 03 00:00:02.521673 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.521654 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerDied","Data":"621588198b060c8f74ce27e7b43df505fd49bd845ce67f529baab8a9ea0d4fad"} Nov 03 00:00:02.521849 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.521833 4494 scope.go:122] "RemoveContainer" containerID="621588198b060c8f74ce27e7b43df505fd49bd845ce67f529baab8a9ea0d4fad" Nov 03 00:00:02.523285 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.523261 4494 generic.go:356] "Generic (PLEG): container finished" podID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerID="310f488e4e1c3218a2314c56de4b52a65a5c8c81805e965fbe2f5dc9dd1ba971" exitCode=60 Nov 03 00:00:02.523385 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.523334 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerDied","Data":"310f488e4e1c3218a2314c56de4b52a65a5c8c81805e965fbe2f5dc9dd1ba971"} Nov 03 00:00:02.523526 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:02.523510 4494 scope.go:122] "RemoveContainer" containerID="310f488e4e1c3218a2314c56de4b52a65a5c8c81805e965fbe2f5dc9dd1ba971" Nov 03 00:00:03.535436 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.534451 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerStarted","Data":"bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2"} Nov 03 00:00:03.546253 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.539341 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerStarted","Data":"4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b"} Nov 03 00:00:03.546253 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.542562 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerStarted","Data":"705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a"} Nov 03 00:00:03.546253 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.544919 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerStarted","Data":"7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e"} Nov 03 00:00:03.547467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.547442 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerStarted","Data":"1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9"} Nov 03 00:00:03.547757 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.547744 4494 scope.go:122] "RemoveContainer" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" Nov 03 00:00:03.548020 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:03.548003 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr_odh-ai-gateway-infra(499e1125-ce2a-4007-bcf5-6c6ceaf37ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" Nov 03 00:00:03.767825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.767771 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" podStartSLOduration=3.767753394 podStartE2EDuration="3.767753394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.766569286 +0000 UTC m=+4318.196439147" watchObservedRunningTime="2026-11-03 00:00:03.767753394 +0000 UTC m=+4318.197623256" Nov 03 00:00:03.887147 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:03.887091 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" podStartSLOduration=3.887076436 podStartE2EDuration="3.887076436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.885853492 +0000 UTC m=+4318.315723354" watchObservedRunningTime="2026-11-03 00:00:03.887076436 +0000 UTC m=+4318.316946348" Nov 03 00:00:04.121610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.121509 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" podStartSLOduration=4.121495489 podStartE2EDuration="4.121495489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.118921282 +0000 UTC m=+4318.548791143" watchObservedRunningTime="2026-11-03 00:00:04.121495489 +0000 UTC m=+4318.551365350" Nov 03 00:00:04.556800 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.556759 4494 generic.go:356] "Generic (PLEG): container finished" podID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" exitCode=60 Nov 03 00:00:04.557301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.556820 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerDied","Data":"1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9"} Nov 03 00:00:04.557301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.556883 4494 scope.go:122] "RemoveContainer" containerID="86d9e84bbca8bcc3d7384e3b324ad2fd1a116e50a178a2645d62fe31f2f87850" Nov 03 00:00:04.557301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.557190 4494 scope.go:122] "RemoveContainer" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" Nov 03 00:00:04.557612 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:04.557581 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr_odh-ai-gateway-infra(499e1125-ce2a-4007-bcf5-6c6ceaf37ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" Nov 03 00:00:04.568604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.568550 4494 generic.go:356] "Generic (PLEG): container finished" podID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerID="bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2" exitCode=60 Nov 03 00:00:04.568763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.568703 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerDied","Data":"bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2"} Nov 03 00:00:04.569558 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.569456 4494 scope.go:122] "RemoveContainer" containerID="bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2" Nov 03 00:00:04.570147 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:04.570109 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-pq625_odh-ai-gateway-infra(2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" Nov 03 00:00:04.571911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.571871 4494 generic.go:356] "Generic (PLEG): container finished" podID="58f4f435-8294-4190-840e-43e1da0caf9e" containerID="4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b" exitCode=60 Nov 03 00:00:04.572040 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.572016 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerDied","Data":"4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b"} Nov 03 00:00:04.572267 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.572248 4494 scope.go:122] "RemoveContainer" containerID="4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b" Nov 03 00:00:04.572729 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:04.572702 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6_odh-ai-gateway-infra(58f4f435-8294-4190-840e-43e1da0caf9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" Nov 03 00:00:04.574531 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.574509 4494 generic.go:356] "Generic (PLEG): container finished" podID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerID="705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a" exitCode=60 Nov 03 00:00:04.574656 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.574632 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerDied","Data":"705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a"} Nov 03 00:00:04.575094 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.575076 4494 scope.go:122] "RemoveContainer" containerID="705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a" Nov 03 00:00:04.575379 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:04.575354 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g_odh-ai-gateway-infra(2f5e7357-60d7-407b-89c1-6f4b1ee64dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" Nov 03 00:00:04.576838 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.576814 4494 generic.go:356] "Generic (PLEG): container finished" podID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerID="7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e" exitCode=60 Nov 03 00:00:04.576982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.576952 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerDied","Data":"7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e"} Nov 03 00:00:04.577131 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.577117 4494 scope.go:122] "RemoveContainer" containerID="7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e" Nov 03 00:00:04.577400 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:04.577376 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb_odh-ai-gateway-infra(9007ede0-103c-4ea4-9933-43d4a5b74583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" Nov 03 00:00:04.583345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.583324 4494 scope.go:122] "RemoveContainer" containerID="cba30fd36bbca136fec0e4d5d4f6e18215951e85fa740fbcb0c73c19d0ee83a6" Nov 03 00:00:04.596955 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.596931 4494 scope.go:122] "RemoveContainer" containerID="646750914a0f84e2136a6eef586ad25af05f1a24baf8b96ebc7014e5a3cf5d0a" Nov 03 00:00:04.610835 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.610813 4494 scope.go:122] "RemoveContainer" containerID="621588198b060c8f74ce27e7b43df505fd49bd845ce67f529baab8a9ea0d4fad" Nov 03 00:00:04.624074 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:04.624052 4494 scope.go:122] "RemoveContainer" containerID="310f488e4e1c3218a2314c56de4b52a65a5c8c81805e965fbe2f5dc9dd1ba971" Nov 03 00:00:05.240630 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.240584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:05.585342 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.585253 4494 scope.go:122] "RemoveContainer" containerID="705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a" Nov 03 00:00:05.585809 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:05.585591 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g_odh-ai-gateway-infra(2f5e7357-60d7-407b-89c1-6f4b1ee64dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" Nov 03 00:00:05.587005 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.586987 4494 scope.go:122] "RemoveContainer" containerID="7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e" Nov 03 00:00:05.587281 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:05.587260 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb_odh-ai-gateway-infra(9007ede0-103c-4ea4-9933-43d4a5b74583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" Nov 03 00:00:05.588738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.588718 4494 scope.go:122] "RemoveContainer" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" Nov 03 00:00:05.588947 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:05.588922 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr_odh-ai-gateway-infra(499e1125-ce2a-4007-bcf5-6c6ceaf37ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" Nov 03 00:00:05.590565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.590335 4494 scope.go:122] "RemoveContainer" containerID="bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2" Nov 03 00:00:05.590674 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:05.590573 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-pq625_odh-ai-gateway-infra(2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" Nov 03 00:00:05.592007 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:05.591991 4494 scope.go:122] "RemoveContainer" containerID="4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b" Nov 03 00:00:05.592202 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:05.592162 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6_odh-ai-gateway-infra(58f4f435-8294-4190-840e-43e1da0caf9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" Nov 03 00:00:06.248251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:06.248200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:10.128340 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:10.128230 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:11.757413 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:11.757380 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:00:14.694194 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:14.694137 4494 scope.go:122] "RemoveContainer" containerID="bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2" Nov 03 00:00:15.112723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:15.112678 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:15.690964 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:15.690936 4494 scope.go:122] "RemoveContainer" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" Nov 03 00:00:15.725430 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:15.725399 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerStarted","Data":"7065aac6e0315f354ffd9885ceada14a9145cc577810e8c12ec8a5c1a0e67ac8"} Nov 03 00:00:16.686547 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.686522 4494 scope.go:122] "RemoveContainer" containerID="705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a" Nov 03 00:00:16.686817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.686800 4494 scope.go:122] "RemoveContainer" containerID="4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b" Nov 03 00:00:16.686897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.686864 4494 scope.go:122] "RemoveContainer" containerID="7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e" Nov 03 00:00:16.735450 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.735418 4494 generic.go:356] "Generic (PLEG): container finished" podID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerID="feba550ac965021f5d767b060e111c971e448d2b2a644e21d78283d268119876" exitCode=60 Nov 03 00:00:16.735905 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.735491 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerDied","Data":"feba550ac965021f5d767b060e111c971e448d2b2a644e21d78283d268119876"} Nov 03 00:00:16.735905 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.735550 4494 scope.go:122] "RemoveContainer" containerID="1782e91259866591a1c72529faeb333ee3149e3263293ccc43c3eec0918d36e9" Nov 03 00:00:16.736021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.735998 4494 scope.go:122] "RemoveContainer" containerID="feba550ac965021f5d767b060e111c971e448d2b2a644e21d78283d268119876" Nov 03 00:00:16.736390 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:16.736363 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr_odh-ai-gateway-infra(499e1125-ce2a-4007-bcf5-6c6ceaf37ee7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" Nov 03 00:00:16.737935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.737907 4494 generic.go:356] "Generic (PLEG): container finished" podID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerID="7065aac6e0315f354ffd9885ceada14a9145cc577810e8c12ec8a5c1a0e67ac8" exitCode=60 Nov 03 00:00:16.738056 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.737985 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerDied","Data":"7065aac6e0315f354ffd9885ceada14a9145cc577810e8c12ec8a5c1a0e67ac8"} Nov 03 00:00:16.738425 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.738400 4494 scope.go:122] "RemoveContainer" containerID="7065aac6e0315f354ffd9885ceada14a9145cc577810e8c12ec8a5c1a0e67ac8" Nov 03 00:00:16.738721 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:16.738691 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-pq625_odh-ai-gateway-infra(2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" Nov 03 00:00:16.754209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.754162 4494 scope.go:122] "RemoveContainer" containerID="bd0c652dac8cb6897b022b2ad9c6b8b5fa84c8450c94d492ca0c1f38336d17f2" Nov 03 00:00:16.908431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:16.908359 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:17.563224 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:17.563184 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:00:17.746762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.746730 4494 generic.go:356] "Generic (PLEG): container finished" podID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerID="789ec84081c08fbb57567939b19e9b910c1929ee3e3377b85b0cf5f3b5cf625c" exitCode=60 Nov 03 00:00:17.747151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.746809 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerDied","Data":"789ec84081c08fbb57567939b19e9b910c1929ee3e3377b85b0cf5f3b5cf625c"} Nov 03 00:00:17.747151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.746862 4494 scope.go:122] "RemoveContainer" containerID="705fcea0e9093a31704a58738ed23cefda4aca6205ac3d9ef8e77b0dad71f15a" Nov 03 00:00:17.747356 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.747337 4494 scope.go:122] "RemoveContainer" containerID="789ec84081c08fbb57567939b19e9b910c1929ee3e3377b85b0cf5f3b5cf625c" Nov 03 00:00:17.747735 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:17.747708 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g_odh-ai-gateway-infra(2f5e7357-60d7-407b-89c1-6f4b1ee64dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" Nov 03 00:00:17.748853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.748832 4494 generic.go:356] "Generic (PLEG): container finished" podID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerID="2aa9bb75e55c95c43eaa07a4a0ed7b4207ecd2596806d277f603b16446847059" exitCode=60 Nov 03 00:00:17.748953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.748894 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerDied","Data":"2aa9bb75e55c95c43eaa07a4a0ed7b4207ecd2596806d277f603b16446847059"} Nov 03 00:00:17.749303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.749282 4494 scope.go:122] "RemoveContainer" containerID="2aa9bb75e55c95c43eaa07a4a0ed7b4207ecd2596806d277f603b16446847059" Nov 03 00:00:17.749598 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:17.749575 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb_odh-ai-gateway-infra(9007ede0-103c-4ea4-9933-43d4a5b74583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" Nov 03 00:00:17.754962 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.754933 4494 generic.go:356] "Generic (PLEG): container finished" podID="58f4f435-8294-4190-840e-43e1da0caf9e" containerID="8fcf0b27ca7eda60d464e74202eed7b3af38d72fdccb72fbefd6c1e2a6e609cb" exitCode=60 Nov 03 00:00:17.755066 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.754996 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerDied","Data":"8fcf0b27ca7eda60d464e74202eed7b3af38d72fdccb72fbefd6c1e2a6e609cb"} Nov 03 00:00:17.755334 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.755318 4494 scope.go:122] "RemoveContainer" containerID="8fcf0b27ca7eda60d464e74202eed7b3af38d72fdccb72fbefd6c1e2a6e609cb" Nov 03 00:00:17.755579 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:17.755561 4494 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6_odh-ai-gateway-infra(58f4f435-8294-4190-840e-43e1da0caf9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" Nov 03 00:00:17.831086 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.831051 4494 scope.go:122] "RemoveContainer" containerID="7840862d9570c4c236657801c6f820aa96cffcee73f4923f9c0e00764f5fef4e" Nov 03 00:00:17.848056 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.848029 4494 scope.go:122] "RemoveContainer" containerID="4bce24567df775ba96644ee05226a4f2ea199ccd47501a07ec0bf5a08325e56b" Nov 03 00:00:17.875754 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.875721 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:17.914217 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.914188 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:17.959355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.959325 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\" (UniqueName: \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\") pod \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " Nov 03 00:00:17.959517 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.959456 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\") pod \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\" (UID: \"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8\") " Nov 03 00:00:17.959868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.959838 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca" pod "2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" (UID: "2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:17.960303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.960280 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:17.961879 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:17.961856 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs" pod "2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" (UID: "2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8"). InnerVolumeSpecName "kube-api-access-k5nhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.019345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.019318 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:18.061344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.061256 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\" (UniqueName: \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\") pod \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " Nov 03 00:00:18.061537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.061397 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\") pod \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\" (UID: \"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7\") " Nov 03 00:00:18.061722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.061708 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5nhs\" (UniqueName: \"kubernetes.io/projected/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8-kube-api-access-k5nhs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.061790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.061713 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca" pod "499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" (UID: "499e1125-ce2a-4007-bcf5-6c6ceaf37ee7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.063552 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.063532 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9" pod "499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" (UID: "499e1125-ce2a-4007-bcf5-6c6ceaf37ee7"). InnerVolumeSpecName "kube-api-access-6q9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.163075 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.163046 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.163075 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.163073 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9x9\" (UniqueName: \"kubernetes.io/projected/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7-kube-api-access-6q9x9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.768637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.768609 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" Nov 03 00:00:18.768637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.768616 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625" event={"ID":"2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8","Type":"ContainerDied","Data":"ed2178d2c782ae20512ae6a104baa1bc47bab4f480f5f6a029ff9412143b69aa"} Nov 03 00:00:18.769211 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.768716 4494 scope.go:122] "RemoveContainer" containerID="7065aac6e0315f354ffd9885ceada14a9145cc577810e8c12ec8a5c1a0e67ac8" Nov 03 00:00:18.775977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.775949 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" event={"ID":"499e1125-ce2a-4007-bcf5-6c6ceaf37ee7","Type":"ContainerDied","Data":"1eb3f9aad3d4e15acf30499ed19055cfce01ca87d165dc37baa192b8f163bdc7"} Nov 03 00:00:18.776112 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.776024 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr" Nov 03 00:00:18.784320 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.784105 4494 scope.go:122] "RemoveContainer" containerID="feba550ac965021f5d767b060e111c971e448d2b2a644e21d78283d268119876" Nov 03 00:00:18.878981 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.878904 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:18.938626 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.938598 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:18.950613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.950584 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b8d3cb-29894400-lvdmr"] Nov 03 00:00:18.995600 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:18.995570 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:19.041478 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.041450 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:19.061930 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.061897 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:19.069694 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.067350 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-pq625"] Nov 03 00:00:19.077823 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.077790 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\") pod \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " Nov 03 00:00:19.077992 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.077858 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\" (UniqueName: \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\") pod \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\" (UID: \"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7\") " Nov 03 00:00:19.078251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.078224 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca" pod "2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" (UID: "2f5e7357-60d7-407b-89c1-6f4b1ee64dc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.078613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.078590 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.081142 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.081118 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv" pod "2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" (UID: "2f5e7357-60d7-407b-89c1-6f4b1ee64dc7"). InnerVolumeSpecName "kube-api-access-2sdnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.113283 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.113256 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:19.169716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.169665 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:19.183564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.183537 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdnv\" (UniqueName: \"kubernetes.io/projected/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7-kube-api-access-2sdnv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.260574 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.260549 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:19.284550 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.284526 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\" (UniqueName: \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\") pod \"58f4f435-8294-4190-840e-43e1da0caf9e\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " Nov 03 00:00:19.284673 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.284560 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\") pod \"58f4f435-8294-4190-840e-43e1da0caf9e\" (UID: \"58f4f435-8294-4190-840e-43e1da0caf9e\") " Nov 03 00:00:19.284945 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.284924 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca" pod "58f4f435-8294-4190-840e-43e1da0caf9e" (UID: "58f4f435-8294-4190-840e-43e1da0caf9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.286730 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.286710 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48" pod "58f4f435-8294-4190-840e-43e1da0caf9e" (UID: "58f4f435-8294-4190-840e-43e1da0caf9e"). InnerVolumeSpecName "kube-api-access-4cl48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.386264 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.386234 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\") pod \"9007ede0-103c-4ea4-9933-43d4a5b74583\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " Nov 03 00:00:19.386497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.386318 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\" (UniqueName: \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\") pod \"9007ede0-103c-4ea4-9933-43d4a5b74583\" (UID: \"9007ede0-103c-4ea4-9933-43d4a5b74583\") " Nov 03 00:00:19.386652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.386617 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca" pod "9007ede0-103c-4ea4-9933-43d4a5b74583" (UID: "9007ede0-103c-4ea4-9933-43d4a5b74583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.386727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.386652 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cl48\" (UniqueName: \"kubernetes.io/projected/58f4f435-8294-4190-840e-43e1da0caf9e-kube-api-access-4cl48\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.386727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.386671 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f4f435-8294-4190-840e-43e1da0caf9e-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.388533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.388510 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml" pod "9007ede0-103c-4ea4-9933-43d4a5b74583" (UID: "9007ede0-103c-4ea4-9933-43d4a5b74583"). InnerVolumeSpecName "kube-api-access-j5qml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.487980 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.487933 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9007ede0-103c-4ea4-9933-43d4a5b74583-maas-api-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.487980 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.487971 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5qml\" (UniqueName: \"kubernetes.io/projected/9007ede0-103c-4ea4-9933-43d4a5b74583-kube-api-access-j5qml\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.695502 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.695467 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" path="/var/lib/kubelet/pods/2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8/volumes" Nov 03 00:00:19.696030 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.696007 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" path="/var/lib/kubelet/pods/499e1125-ce2a-4007-bcf5-6c6ceaf37ee7/volumes" Nov 03 00:00:19.784685 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.784601 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" Nov 03 00:00:19.784685 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.784637 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6" event={"ID":"58f4f435-8294-4190-840e-43e1da0caf9e","Type":"ContainerDied","Data":"57cda572de0168c37e38eb40fe3ef83f12f2d34b201a05d6c2926edad54ae5f7"} Nov 03 00:00:19.785240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.784705 4494 scope.go:122] "RemoveContainer" containerID="8fcf0b27ca7eda60d464e74202eed7b3af38d72fdccb72fbefd6c1e2a6e609cb" Nov 03 00:00:19.786382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.786359 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" Nov 03 00:00:19.786382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.786373 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g" event={"ID":"2f5e7357-60d7-407b-89c1-6f4b1ee64dc7","Type":"ContainerDied","Data":"7378a104ae3b7ebf7e9211e1bcc0055bb53398d49b91bca7f0b79906ddcaef55"} Nov 03 00:00:19.788239 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.788211 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" event={"ID":"9007ede0-103c-4ea4-9933-43d4a5b74583","Type":"ContainerDied","Data":"6b2e5b019ebfdf0a0a5663d38c1d70b864b31a2ff814f4a9271ba0e378edef95"} Nov 03 00:00:19.789221 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.788609 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb" Nov 03 00:00:19.798070 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.798047 4494 scope.go:122] "RemoveContainer" containerID="789ec84081c08fbb57567939b19e9b910c1929ee3e3377b85b0cf5f3b5cf625c" Nov 03 00:00:19.807430 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.807402 4494 scope.go:122] "RemoveContainer" containerID="2aa9bb75e55c95c43eaa07a4a0ed7b4207ecd2596806d277f603b16446847059" Nov 03 00:00:19.953566 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.953534 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:19.957532 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:19.957505 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-637fdb-29894400-pjn8g"] Nov 03 00:00:20.047630 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.047544 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:20.068257 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.068225 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:20.072956 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.072920 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5e7e9e-29894400-hxlx6"] Nov 03 00:00:20.186259 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.186229 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:20.189312 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.189286 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-96031a-29894400-mrdpb"] Nov 03 00:00:20.262560 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:20.262530 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:21.694505 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:21.694474 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" path="/var/lib/kubelet/pods/2f5e7357-60d7-407b-89c1-6f4b1ee64dc7/volumes" Nov 03 00:00:21.695043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:21.695028 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" path="/var/lib/kubelet/pods/58f4f435-8294-4190-840e-43e1da0caf9e/volumes" Nov 03 00:00:21.695474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:21.695459 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" path="/var/lib/kubelet/pods/9007ede0-103c-4ea4-9933-43d4a5b74583/volumes" Nov 03 00:00:24.934057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:24.934019 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:29.856063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:29.856028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:34.751919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:34.751882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:39.740007 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:39.739966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:44.721196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:44.712745 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:47.570907 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:47.570817 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:00:48.925787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:48.925751 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:50.306343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:50.306308 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:50.452699 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:50.452657 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:50.611392 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:50.611308 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:50.758918 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:50.758884 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:51.044786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:51.044741 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:51.111378 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:51.111339 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 03 00:00:51.111595 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:51.111562 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" containerID="cri-o://0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44" gracePeriod=30 Nov 03 00:00:51.250196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:51.237099 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:53.394088 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:53.394050 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:53.540656 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:53.540622 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:53.919625 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:53.919597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:55.247189 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.247135 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 03 00:00:55.247682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.247423 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerName="maas-api" containerID="cri-o://b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9" gracePeriod=30 Nov 03 00:00:55.352556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.352526 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:55.568981 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.568954 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 03 00:00:55.708061 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.708028 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\") pod \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " Nov 03 00:00:55.708395 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.708371 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\" (UniqueName: \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\") pod \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " Nov 03 00:00:55.708584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.708567 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\") pod \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " Nov 03 00:00:55.708713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.708695 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\") pod \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\" (UID: \"1c89e234-56b7-432c-9aa3-c4c7b214a7f3\") " Nov 03 00:00:55.709515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.709246 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca" pod "1c89e234-56b7-432c-9aa3-c4c7b214a7f3" (UID: "1c89e234-56b7-432c-9aa3-c4c7b214a7f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:55.712082 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.712053 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs" pod "1c89e234-56b7-432c-9aa3-c4c7b214a7f3" (UID: "1c89e234-56b7-432c-9aa3-c4c7b214a7f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:00:55.712224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.712141 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls" pod "1c89e234-56b7-432c-9aa3-c4c7b214a7f3" (UID: "1c89e234-56b7-432c-9aa3-c4c7b214a7f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:00:55.712304 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.712281 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9" pod "1c89e234-56b7-432c-9aa3-c4c7b214a7f3" (UID: "1c89e234-56b7-432c-9aa3-c4c7b214a7f3"). InnerVolumeSpecName "kube-api-access-n8rm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:55.810346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.810271 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:55.810346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.810299 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rm9\" (UniqueName: \"kubernetes.io/projected/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-kube-api-access-n8rm9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:55.810346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.810311 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:55.810346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:55.810320 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89e234-56b7-432c-9aa3-c4c7b214a7f3-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.035939 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.035905 4494 generic.go:356] "Generic (PLEG): container finished" podID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerID="b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9" exitCode=0 Nov 03 00:00:56.036144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.036001 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" event={"ID":"1c89e234-56b7-432c-9aa3-c4c7b214a7f3","Type":"ContainerDied","Data":"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9"} Nov 03 00:00:56.036144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.036017 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" Nov 03 00:00:56.036144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.036055 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" event={"ID":"1c89e234-56b7-432c-9aa3-c4c7b214a7f3","Type":"ContainerDied","Data":"0cbed8ad0aa448ab942005793dffb0d200c8f02c50661b3b2eb311765453f5a7"} Nov 03 00:00:56.036144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.036085 4494 scope.go:122] "RemoveContainer" containerID="b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9" Nov 03 00:00:56.048288 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.048265 4494 scope.go:122] "RemoveContainer" containerID="b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9" Nov 03 00:00:56.048584 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:56.048560 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9\": container with ID starting with b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9 not found: ID does not exist" containerID="b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9" Nov 03 00:00:56.048687 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.048593 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9"} err="failed to get container status \"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9\": rpc error: code = NotFound desc = could not find container \"b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9\": container with ID starting with b1d191d51fd3eb9513e2993e539a11148248c62cf429bd286b7d6e6f3e61c8e9 not found: ID does not exist" Nov 03 00:00:56.222196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.218228 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 03 00:00:56.227197 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.224272 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg"] Nov 03 00:00:56.446746 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.446722 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523511 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523580 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523627 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523680 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523776 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523823 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523885 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.523978 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.525268 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.524109 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\") pod \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\" (UID: \"2e4ea97d-9907-4240-9f3d-94f7d7f92174\") " Nov 03 00:00:56.528661 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.528468 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:00:56.528661 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.528391 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:56.529309 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.528931 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:00:56.529309 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.529300 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:00:56.529701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.529540 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:00:56.531847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.529948 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:00:56.531968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.531905 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:00:56.538873 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.538809 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "kube-api-access-8lhrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:56.541243 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.541210 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token" pod "2e4ea97d-9907-4240-9f3d-94f7d7f92174" (UID: "2e4ea97d-9907-4240-9f3d-94f7d7f92174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:56.626674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626633 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626668 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626682 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626695 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626708 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626719 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626730 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4ea97d-9907-4240-9f3d-94f7d7f92174-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626744 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4ea97d-9907-4240-9f3d-94f7d7f92174-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:56.626943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:56.626756 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhrf\" (UniqueName: \"kubernetes.io/projected/2e4ea97d-9907-4240-9f3d-94f7d7f92174-kube-api-access-8lhrf\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:00:57.045216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.045163 4494 generic.go:356] "Generic (PLEG): container finished" podID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerID="0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44" exitCode=0 Nov 03 00:00:57.045216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.045198 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerDied","Data":"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44"} Nov 03 00:00:57.045460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.045249 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" event={"ID":"2e4ea97d-9907-4240-9f3d-94f7d7f92174","Type":"ContainerDied","Data":"9c61682a597a3bd98d77772b162407545ffc9c1bfdd35fcc8db17295c6e9a791"} Nov 03 00:00:57.045460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.045254 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp" Nov 03 00:00:57.045460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.045271 4494 scope.go:122] "RemoveContainer" containerID="0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44" Nov 03 00:00:57.055649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.055627 4494 scope.go:122] "RemoveContainer" containerID="b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3" Nov 03 00:00:57.064762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.064745 4494 scope.go:122] "RemoveContainer" containerID="0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44" Nov 03 00:00:57.065067 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:57.065039 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44\": container with ID starting with 0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44 not found: ID does not exist" containerID="0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44" Nov 03 00:00:57.065132 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.065076 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44"} err="failed to get container status \"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44\": rpc error: code = NotFound desc = could not find container \"0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44\": container with ID starting with 0973d08bad40f5f996d1bf34f6653204d6eeceb7bc04ee88b821f407c3253f44 not found: ID does not exist" Nov 03 00:00:57.065132 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.065093 4494 scope.go:122] "RemoveContainer" containerID="b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3" Nov 03 00:00:57.065368 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:00:57.065351 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3\": container with ID starting with b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3 not found: ID does not exist" containerID="b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3" Nov 03 00:00:57.065427 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.065378 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3"} err="failed to get container status \"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3\": rpc error: code = NotFound desc = could not find container \"b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3\": container with ID starting with b9bed020f60acd544cd42b9fdc8bbbae51f9d14f764bf1315eeddec3790b50e3 not found: ID does not exist" Nov 03 00:00:57.151898 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.151864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:57.243728 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.243690 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 03 00:00:57.246940 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.246911 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5db41e-openshift-default-fb5f5b577-7wjpp"] Nov 03 00:00:57.695618 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.695589 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" path="/var/lib/kubelet/pods/1c89e234-56b7-432c-9aa3-c4c7b214a7f3/volumes" Nov 03 00:00:57.696043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:57.696030 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" path="/var/lib/kubelet/pods/2e4ea97d-9907-4240-9f3d-94f7d7f92174/volumes" Nov 03 00:00:58.504610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:58.504568 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5e7e9e-764bff87b7-whvlg" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.30:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:00:58.856056 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:58.855942 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:00:59.004821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:00:59.004787 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.073805 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.073769 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.225762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.225737 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.371255 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.371160 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.522064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.522033 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.666874 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.666839 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:00.909679 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:00.909649 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:01.772735 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:01.772699 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:02.354882 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.354847 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:02.804816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.804777 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:01:02.805537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805520 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805539 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805554 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805560 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805569 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805574 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805586 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.805590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805591 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805600 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805605 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805613 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805617 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805624 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805628 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805634 4494 cpu_manager.go:414] "containerMap: removing container" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805638 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805644 4494 cpu_manager.go:414] "containerMap: removing container" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805647 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805653 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805657 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805667 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805671 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805677 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805682 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805694 4494 cpu_manager.go:414] "containerMap: removing container" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerName="maas-api" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805698 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerName="maas-api" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805707 4494 cpu_manager.go:414] "containerMap: removing container" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805711 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805718 4494 cpu_manager.go:414] "containerMap: removing container" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805723 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805732 4494 cpu_manager.go:414] "containerMap: removing container" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805736 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805749 4494 cpu_manager.go:414] "containerMap: removing container" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805754 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805764 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.805812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805768 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805854 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805863 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805870 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805876 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805882 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805888 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805893 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805898 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805903 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4ea97d-9907-4240-9f3d-94f7d7f92174" containerName="istio-proxy" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805909 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805923 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5e7357-60d7-407b-89c1-6f4b1ee64dc7" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805931 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c89e234-56b7-432c-9aa3-c4c7b214a7f3" containerName="maas-api" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805938 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805944 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.806767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.805951 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="499e1125-ce2a-4007-bcf5-6c6ceaf37ee7" containerName="cleanup" Nov 03 00:01:02.811126 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.811109 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.869237 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.869194 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3bee1b-openshift-default-dockercfg-fhqz8\"" Nov 03 00:01:02.892324 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.892291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:02.924081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924029 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924294 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924108 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bf49\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924294 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924220 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:01:02.924294 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924245 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924344 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924488 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924521 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924537 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924555 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:02.924779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:02.924611 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025760 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025724 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025780 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025829 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025851 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025873 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025899 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.025991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.025934 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026186 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026240 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bf49\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026309 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026318 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026645 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.026881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.026862 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.027039 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.027015 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.028502 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.028472 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.028777 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.028760 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.126509 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.126431 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bf49\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.126509 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.126471 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\") pod \"e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.282720 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.282673 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:03.425182 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.425142 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:03.859934 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.859906 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:01:03.860643 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:01:03.860621 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d294e2f_18e9_478a_99cc_8a307195ad58.slice/crio-1392d1239cb46bfd3ede1782454b9dae9a3f33b528956b213730f63bde3ba8a4 WatchSource:0}: Error finding container 1392d1239cb46bfd3ede1782454b9dae9a3f33b528956b213730f63bde3ba8a4: Status 404 returned error can't find the container with id 1392d1239cb46bfd3ede1782454b9dae9a3f33b528956b213730f63bde3ba8a4 Nov 03 00:01:03.863153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.863119 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:01:03.863315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.863252 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:01:03.863315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:03.863303 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:01:04.104812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.104774 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerStarted","Data":"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819"} Nov 03 00:01:04.105028 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.104833 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerStarted","Data":"1392d1239cb46bfd3ede1782454b9dae9a3f33b528956b213730f63bde3ba8a4"} Nov 03 00:01:04.227907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.227854 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podStartSLOduration=2.227837176 podStartE2EDuration="2.227837176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:04.225314543 +0000 UTC m=+4378.655184407" watchObservedRunningTime="2026-11-03 00:01:04.227837176 +0000 UTC m=+4378.657707027" Nov 03 00:01:04.371215 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.371132 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:04.425539 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.425511 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:04.431089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.431064 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:04.515919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:04.515822 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:05.111684 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:05.111653 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:05.112623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:05.112604 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:01:06.990233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:06.990182 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:07.138331 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:07.138303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:07.286879 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:07.286789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:07.646326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:07.646245 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:10.275544 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:10.275509 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:10.549247 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:10.533578 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:10.698443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:10.698413 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:11.031711 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:11.031677 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:12.948250 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:12.948220 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:14.865679 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:14.865637 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:01:16.844862 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:16.844818 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:17.578358 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:01:17.578303 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:01:18.064909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:18.064858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:20.537243 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:20.537205 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:23.142740 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:23.142697 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:28.030549 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:28.030506 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:30.210921 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:30.210882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:30.423791 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:30.423750 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:31.097094 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:31.097054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:32.976355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:32.976315 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:37.794208 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:37.794160 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:42.671528 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:42.671487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:47.561579 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:01:47.561538 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:01:47.571918 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:47.571889 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:52.468665 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:52.468636 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:57.341937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:57.341894 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:57.486001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:57.485966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:01:58.676925 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:01:58.676888 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:02:02.358844 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:02.358809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:03.161861 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:03.161827 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:07.240808 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:07.240770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:12.233732 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:12.233695 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:16.836099 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.836068 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:02:16.836839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.836823 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba8d1b9-94bd-4885-9f4a-ea24952cb1b8" containerName="cleanup" Nov 03 00:02:16.836886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.836846 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9007ede0-103c-4ea4-9933-43d4a5b74583" containerName="cleanup" Nov 03 00:02:16.836886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.836858 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f4f435-8294-4190-840e-43e1da0caf9e" containerName="cleanup" Nov 03 00:02:16.840361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.840324 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:16.894392 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.894360 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3bee1b-serving-cert\"" Nov 03 00:02:16.948584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:16.948554 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:02:17.022434 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.022402 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:17.027803 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.027778 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.027953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.027902 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.028011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.027983 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.028103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.028093 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5t6\" (UniqueName: \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.132652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.132566 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.132963 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.132938 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5t6\" (UniqueName: \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.133605 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.133577 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.133747 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.133705 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.134052 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:02:17.132749 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-serving-cert: secret "maas-api-e2e-shared-a-w1-3bee1b-serving-cert" not found Nov 03 00:02:17.134155 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:02:17.134060 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls podName:9b10f152-5990-468a-918b-3c6b6ed63e35 nodeName:}" failed. No retries permitted until 2026-11-03 00:02:17.63403572 +0000 UTC m=+4452.063905574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls") pod "maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" (UID: "9b10f152-5990-468a-918b-3c6b6ed63e35") : secret "maas-api-e2e-shared-a-w1-3bee1b-serving-cert" not found Nov 03 00:02:17.134410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.134391 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.145314 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.142886 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.177546 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.177520 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:17.196139 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.196106 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5t6\" (UniqueName: \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.509843 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.509807 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:17.560435 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:02:17.560354 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:02:17.645719 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.645686 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.648515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.648481 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:17.754486 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:17.754453 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:18.132883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:18.132850 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:02:18.134870 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:02:18.134837 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b10f152_5990_468a_918b_3c6b6ed63e35.slice/crio-0689a5bfd2187a4c1a9d3c352a0b8d581a6d0d26e2ab3f5a01b925cc8e84c612 WatchSource:0}: Error finding container 0689a5bfd2187a4c1a9d3c352a0b8d581a6d0d26e2ab3f5a01b925cc8e84c612: Status 404 returned error can't find the container with id 0689a5bfd2187a4c1a9d3c352a0b8d581a6d0d26e2ab3f5a01b925cc8e84c612 Nov 03 00:02:18.136278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:18.136258 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:02:18.631964 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:18.631928 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" event={"ID":"9b10f152-5990-468a-918b-3c6b6ed63e35","Type":"ContainerStarted","Data":"0689a5bfd2187a4c1a9d3c352a0b8d581a6d0d26e2ab3f5a01b925cc8e84c612"} Nov 03 00:02:19.640824 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:19.640777 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" event={"ID":"9b10f152-5990-468a-918b-3c6b6ed63e35","Type":"ContainerStarted","Data":"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b"} Nov 03 00:02:19.641263 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:19.640863 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:19.761520 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:19.761471 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" podStartSLOduration=2.826206515 podStartE2EDuration="3.761454559s" totalImagesPullingTime="935.248044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:19.75974365 +0000 UTC m=+4454.189613510" watchObservedRunningTime="2026-11-03 00:02:19.761454559 +0000 UTC m=+4454.191324414" Nov 03 00:02:20.879763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:20.879728 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:22.486424 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:22.486393 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:23.632205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:23.632144 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:25.653691 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:25.653660 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:02:27.001048 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:27.001010 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:31.971199 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:31.971145 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:35.041300 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:35.041269 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:35.663702 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:35.663668 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:36.144937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:36.144854 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:36.837647 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:36.837617 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:02:36.852001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:36.851976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:39.814856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:39.814815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:39.986012 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:39.985981 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:40.146375 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:40.146276 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:42.345198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:42.343644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:43.056817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.056782 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:43.228742 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.228702 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:43.778358 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.777610 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:02:43.782384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.782318 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:43.826376 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.826342 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:02:43.869604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.869565 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:43.869796 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.869653 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwc6\" (UniqueName: \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:43.873090 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.873057 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04713b77-dockercfg-bzgnd\"" Nov 03 00:02:43.873090 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.873057 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04713b77\"" Nov 03 00:02:43.971340 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.971300 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:43.971569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.971363 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwc6\" (UniqueName: \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:43.972081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:43.972059 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:44.033864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.033778 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwc6\" (UniqueName: \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\") pod \"payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:44.098551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.098516 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:44.481915 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.481879 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:02:44.483437 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:02:44.483411 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3b9acd_bad8_4f16_8c14_91286a4f3809.slice/crio-09b9b319b4d42b3eb3fc2ba295d4abbc0850c1936688c1994e8aabb47e3d6d65 WatchSource:0}: Error finding container 09b9b319b4d42b3eb3fc2ba295d4abbc0850c1936688c1994e8aabb47e3d6d65: Status 404 returned error can't find the container with id 09b9b319b4d42b3eb3fc2ba295d4abbc0850c1936688c1994e8aabb47e3d6d65 Nov 03 00:02:44.908946 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.908855 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" event={"ID":"5f3b9acd-bad8-4f16-8c14-91286a4f3809","Type":"ContainerStarted","Data":"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f"} Nov 03 00:02:44.908946 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.908911 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" event={"ID":"5f3b9acd-bad8-4f16-8c14-91286a4f3809","Type":"ContainerStarted","Data":"09b9b319b4d42b3eb3fc2ba295d4abbc0850c1936688c1994e8aabb47e3d6d65"} Nov 03 00:02:44.908946 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:44.908943 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:45.039956 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:45.039895 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" podStartSLOduration=2.039880536 podStartE2EDuration="2.039880536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:45.036641782 +0000 UTC m=+4479.466511643" watchObservedRunningTime="2026-11-03 00:02:45.039880536 +0000 UTC m=+4479.469750397" Nov 03 00:02:47.561406 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:02:47.561368 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:02:47.891498 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.890665 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:47.931326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.931289 4494 generic.go:356] "Generic (PLEG): container finished" podID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerID="fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819" exitCode=0 Nov 03 00:02:47.931543 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.931335 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerDied","Data":"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819"} Nov 03 00:02:47.931805 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.931788 4494 scope.go:122] "RemoveContainer" containerID="fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819" Nov 03 00:02:47.933464 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.933440 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:47.933530 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.933513 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:47.933568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:47.933544 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:02:48.955515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:48.955475 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerStarted","Data":"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24"} Nov 03 00:02:49.425599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:49.425570 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:02:49.426641 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:49.426620 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 00:02:49.426746 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:49.426664 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 00:02:50.111568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:50.111539 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:02:50.426285 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:50.426251 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 00:02:50.426470 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:50.426326 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 00:02:51.426218 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:51.426157 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 00:02:51.426623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:51.426246 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 00:02:52.425825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:52.425785 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 00:02:52.426022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:52.425859 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 00:02:53.056742 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:53.056683 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:53.426162 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:53.426128 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" start-of-body= Nov 03 00:02:53.426357 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:53.426236 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.37:15021/healthz/ready\": dial tcp 10.133.1.37:15021: connect: connection refused" Nov 03 00:02:54.430210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:54.430155 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:02:54.431282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:54.431261 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:02:55.918372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:55.918341 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:02:56.831434 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:56.831405 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:02:57.962203 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:02:57.962140 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:02.847263 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:02.847217 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:07.756778 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:07.756735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:12.630644 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:12.630611 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:17.546942 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:17.546901 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:17.562700 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:03:17.562659 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:03:21.222077 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:21.222039 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:22.411164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:22.411127 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:27.296462 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:27.296431 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:28.118282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:28.118249 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:32.230449 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:32.230392 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:34.363471 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:34.363430 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:34.768135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:34.768086 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 03 00:03:35.026238 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:35.026140 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:35.168560 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:35.168528 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:37.561429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:37.561397 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:38.588798 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:38.588763 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:40.240413 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:40.240379 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:40.291919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:40.291879 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" containerID="cri-o://61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594" gracePeriod=25 Nov 03 00:03:40.558780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:40.558674 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:41.003004 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.002967 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:41.734036 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.734001 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:03:41.738639 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.738623 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.823634 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.823605 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9b6c9e-openshift-default-dockercfg-76vl8\"" Nov 03 00:03:41.841387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.841355 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:41.851474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.851210 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:03:41.861398 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861373 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.861563 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861438 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.861708 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861642 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.861817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861788 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.861971 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861906 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.861971 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.861947 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.862106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.862034 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.862205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.862122 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtww\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.862385 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.862363 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964017 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964081 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964108 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964130 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964231 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtww\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964259 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964343 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964408 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964461 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964587 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964778 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964727 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964778 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964766 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.964851 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.964783 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.965230 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.965208 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.966826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.966802 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:41.967026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:41.966976 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:42.053190 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.053058 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:42.056431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.054184 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:42.056431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.055108 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtww\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\") pod \"e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:42.286095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.286047 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:42.354291 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.354209 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:42.788771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.788740 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:03:42.789299 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:03:42.789270 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ac90de_2850_48b9_bb53_f4f73c6911af.slice/crio-aadd36dc5305db6eca9927caaf9101c8f1d05a90e90eba7a3c440e321c9d1345 WatchSource:0}: Error finding container aadd36dc5305db6eca9927caaf9101c8f1d05a90e90eba7a3c440e321c9d1345: Status 404 returned error can't find the container with id aadd36dc5305db6eca9927caaf9101c8f1d05a90e90eba7a3c440e321c9d1345 Nov 03 00:03:42.792070 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.792037 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:03:42.792199 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.792114 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:03:42.792280 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.792164 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:03:42.945371 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:42.945344 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:43.368006 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:43.367962 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerStarted","Data":"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb"} Nov 03 00:03:43.368240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:43.368066 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerStarted","Data":"aadd36dc5305db6eca9927caaf9101c8f1d05a90e90eba7a3c440e321c9d1345"} Nov 03 00:03:43.491960 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:43.491912 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podStartSLOduration=2.4919003330000002 podStartE2EDuration="2.491900333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:03:43.488689106 +0000 UTC m=+4537.918558964" watchObservedRunningTime="2026-11-03 00:03:43.491900333 +0000 UTC m=+4537.921770193" Nov 03 00:03:44.354931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:44.354898 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:44.360574 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:44.360550 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:44.375577 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:44.375554 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:44.376298 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:44.376279 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:03:44.523952 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:44.523922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:45.415420 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:45.415342 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 03 00:03:45.663511 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:45.663482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:45.810935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:45.810859 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:46.395337 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:46.395307 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:46.562896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:46.562855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:46.739560 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:46.739527 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:47.530564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.530527 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 03 00:03:47.530769 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.530737 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" podUID="7d0b840d-1529-48aa-98ae-bff9989f9548" containerName="maas-api" containerID="cri-o://032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966" gracePeriod=30 Nov 03 00:03:47.570265 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:03:47.570167 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:03:47.691354 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.691310 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:47.868875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.868852 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 03 00:03:47.974909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.974870 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\") pod \"7d0b840d-1529-48aa-98ae-bff9989f9548\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " Nov 03 00:03:47.975117 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.974946 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\" (UniqueName: \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\") pod \"7d0b840d-1529-48aa-98ae-bff9989f9548\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " Nov 03 00:03:47.975117 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.975089 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\") pod \"7d0b840d-1529-48aa-98ae-bff9989f9548\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " Nov 03 00:03:47.975303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.975252 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\") pod \"7d0b840d-1529-48aa-98ae-bff9989f9548\" (UID: \"7d0b840d-1529-48aa-98ae-bff9989f9548\") " Nov 03 00:03:47.975506 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.975477 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca" pod "7d0b840d-1529-48aa-98ae-bff9989f9548" (UID: "7d0b840d-1529-48aa-98ae-bff9989f9548"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:03:47.975723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.975704 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0b840d-1529-48aa-98ae-bff9989f9548-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:47.977310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.977285 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb" pod "7d0b840d-1529-48aa-98ae-bff9989f9548" (UID: "7d0b840d-1529-48aa-98ae-bff9989f9548"). InnerVolumeSpecName "kube-api-access-7fbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:03:47.977475 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.977458 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs" pod "7d0b840d-1529-48aa-98ae-bff9989f9548" (UID: "7d0b840d-1529-48aa-98ae-bff9989f9548"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:03:47.977538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:47.977484 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls" pod "7d0b840d-1529-48aa-98ae-bff9989f9548" (UID: "7d0b840d-1529-48aa-98ae-bff9989f9548"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:03:48.077458 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.077378 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:48.077458 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.077408 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d0b840d-1529-48aa-98ae-bff9989f9548-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:48.077458 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.077422 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbcb\" (UniqueName: \"kubernetes.io/projected/7d0b840d-1529-48aa-98ae-bff9989f9548-kube-api-access-7fbcb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:03:48.411482 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.411444 4494 generic.go:356] "Generic (PLEG): container finished" podID="7d0b840d-1529-48aa-98ae-bff9989f9548" containerID="032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966" exitCode=0 Nov 03 00:03:48.411681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.411506 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" Nov 03 00:03:48.411681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.411519 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" event={"ID":"7d0b840d-1529-48aa-98ae-bff9989f9548","Type":"ContainerDied","Data":"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966"} Nov 03 00:03:48.411681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.411599 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt" event={"ID":"7d0b840d-1529-48aa-98ae-bff9989f9548","Type":"ContainerDied","Data":"9a6b69a2c64502df3db8a2262c3c59331e67f7bd03cc7f1637009714d7836135"} Nov 03 00:03:48.411681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.411622 4494 scope.go:122] "RemoveContainer" containerID="032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966" Nov 03 00:03:48.422986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.422968 4494 scope.go:122] "RemoveContainer" containerID="032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966" Nov 03 00:03:48.423299 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:03:48.423280 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966\": container with ID starting with 032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966 not found: ID does not exist" containerID="032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966" Nov 03 00:03:48.423359 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.423308 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966"} err="failed to get container status \"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966\": rpc error: code = NotFound desc = could not find container \"032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966\": container with ID starting with 032776f78c7d464736fd48da3a378f60338db3c38d351e10dbcba80f8d988966 not found: ID does not exist" Nov 03 00:03:48.579241 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.579201 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 03 00:03:48.680594 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:48.680499 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f46b61-775fb6b856-cgsnt"] Nov 03 00:03:49.028039 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:49.027943 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:49.693819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:49.693786 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0b840d-1529-48aa-98ae-bff9989f9548" path="/var/lib/kubelet/pods/7d0b840d-1529-48aa-98ae-bff9989f9548/volumes" Nov 03 00:03:49.997735 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:49.997631 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:50.144552 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:50.144519 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:50.415752 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:50.415632 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" containerID="cri-o://a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754" gracePeriod=25 Nov 03 00:03:50.854369 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:50.854288 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:51.011236 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:51.011201 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:51.156534 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:51.156439 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:51.439642 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:51.439608 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:52.285751 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:52.285697 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:54.990740 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:54.990700 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:03:55.509827 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:55.509789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:55.659000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:55.658962 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:56.197143 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:56.197109 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:57.381772 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:57.381738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:57.829602 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:57.829562 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 03 00:03:57.829936 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:57.829874 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" containerID="cri-o://f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3" gracePeriod=30 Nov 03 00:03:57.865551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:57.865516 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:58.112001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:58.111884 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:58.269832 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:58.269798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:03:58.606116 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:03:58.606081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:00.564792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.564763 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 03 00:04:00.641164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.641129 4494 generic.go:356] "Generic (PLEG): container finished" podID="20f1c709-ac51-46d8-923c-0b92234ca076" containerID="61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594" exitCode=0 Nov 03 00:04:00.641411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.641215 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" Nov 03 00:04:00.641411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.641210 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" event={"ID":"20f1c709-ac51-46d8-923c-0b92234ca076","Type":"ContainerDied","Data":"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594"} Nov 03 00:04:00.641411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.641284 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh" event={"ID":"20f1c709-ac51-46d8-923c-0b92234ca076","Type":"ContainerDied","Data":"d1f740ad17907cdc0d341070c189c6e8bfb4b5f7e607bf7bb8c529e058884070"} Nov 03 00:04:00.641411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.641321 4494 scope.go:122] "RemoveContainer" containerID="61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594" Nov 03 00:04:00.651770 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.651744 4494 scope.go:122] "RemoveContainer" containerID="61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594" Nov 03 00:04:00.652065 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:00.652042 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594\": container with ID starting with 61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594 not found: ID does not exist" containerID="61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594" Nov 03 00:04:00.652145 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.652079 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594"} err="failed to get container status \"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594\": rpc error: code = NotFound desc = could not find container \"61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594\": container with ID starting with 61a7328bb933cf7d69601dda977a2e8de251d3c61f1ddad869ddbe54a9708594 not found: ID does not exist" Nov 03 00:04:00.734115 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.734036 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\") pod \"20f1c709-ac51-46d8-923c-0b92234ca076\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " Nov 03 00:04:00.734115 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.734088 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\" (UniqueName: \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\") pod \"20f1c709-ac51-46d8-923c-0b92234ca076\" (UID: \"20f1c709-ac51-46d8-923c-0b92234ca076\") " Nov 03 00:04:00.734493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.734464 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume" pod "20f1c709-ac51-46d8-923c-0b92234ca076" (UID: "20f1c709-ac51-46d8-923c-0b92234ca076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:00.736477 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.736457 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj" pod "20f1c709-ac51-46d8-923c-0b92234ca076" (UID: "20f1c709-ac51-46d8-923c-0b92234ca076"). InnerVolumeSpecName "kube-api-access-jh2fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:00.835809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.835768 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f1c709-ac51-46d8-923c-0b92234ca076-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:00.835809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:00.835804 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2fj\" (UniqueName: \"kubernetes.io/projected/20f1c709-ac51-46d8-923c-0b92234ca076-kube-api-access-jh2fj\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:01.064765 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:01.064690 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 03 00:04:01.068734 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:01.068697 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-440c60-5fcf6ff8d7-x59zh"] Nov 03 00:04:01.693161 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:01.693134 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:01.694802 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:01.694780 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" path="/var/lib/kubelet/pods/20f1c709-ac51-46d8-923c-0b92234ca076/volumes" Nov 03 00:04:02.805096 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:02.805061 4494 patch_prober.go:29] interesting pod/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:04:02.805567 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:02.805115 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:04:03.108535 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.108510 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 03 00:04:03.164924 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.164891 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165104 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.164942 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165104 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.164980 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165104 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165017 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165104 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165055 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165104 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165104 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165130 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165250 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165399 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\") pod \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\" (UID: \"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a\") " Nov 03 00:04:03.165461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165399 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:03.165461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165418 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:03.165772 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165694 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:03.165836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165801 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:03.166076 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165928 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.166076 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165964 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.166076 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.165985 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.166076 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.166003 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.166076 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.166030 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:03.167442 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.167418 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:04:03.167570 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.167538 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:04:03.167649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.167633 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:03.168036 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.168014 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp" pod "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" (UID: "c981b8ca-3c85-4e43-9bfb-d4e2398dec7a"). InnerVolumeSpecName "kube-api-access-fzgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:03.267890 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.267851 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgrp\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-kube-api-access-fzgrp\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.267890 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.267890 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.268137 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.267906 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.268137 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.267921 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.268137 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.267935 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:03.663528 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.663490 4494 generic.go:356] "Generic (PLEG): container finished" podID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerID="f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3" exitCode=0 Nov 03 00:04:03.663775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.663546 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" Nov 03 00:04:03.663775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.663585 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerDied","Data":"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3"} Nov 03 00:04:03.663775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.663654 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm" event={"ID":"c981b8ca-3c85-4e43-9bfb-d4e2398dec7a","Type":"ContainerDied","Data":"25e5ee3ecf5c715cb84fd6d1012da400fc646a4b60d8b53d3f74f58ed3f6dc11"} Nov 03 00:04:03.663775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.663691 4494 scope.go:122] "RemoveContainer" containerID="f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3" Nov 03 00:04:03.674643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.674625 4494 scope.go:122] "RemoveContainer" containerID="ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e" Nov 03 00:04:03.684359 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.684346 4494 scope.go:122] "RemoveContainer" containerID="f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3" Nov 03 00:04:03.684656 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:03.684638 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3\": container with ID starting with f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3 not found: ID does not exist" containerID="f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3" Nov 03 00:04:03.684741 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.684668 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3"} err="failed to get container status \"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3\": rpc error: code = NotFound desc = could not find container \"f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3\": container with ID starting with f3400779423e67c7f2e0b96ed37e320673b42bb79483ef4bdca5a3b5d921e6e3 not found: ID does not exist" Nov 03 00:04:03.684741 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.684692 4494 scope.go:122] "RemoveContainer" containerID="ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e" Nov 03 00:04:03.684966 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:03.684948 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e\": container with ID starting with ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e not found: ID does not exist" containerID="ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e" Nov 03 00:04:03.685030 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.684972 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e"} err="failed to get container status \"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e\": rpc error: code = NotFound desc = could not find container \"ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e\": container with ID starting with ea30e1cbebe057e85560f737b38f64120acca320cab6e02b7dbde4ddf51f542e not found: ID does not exist" Nov 03 00:04:03.833063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.833030 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 03 00:04:03.884105 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.884070 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-440c60-openshift-default-6d47ffb577-x6rtm"] Nov 03 00:04:03.976886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:03.976804 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 03 00:04:04.194618 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:04.194582 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:04.443934 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:04.443899 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:04.733446 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:04.733364 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:04.991117 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:04.991018 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:05.715917 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:05.715885 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" path="/var/lib/kubelet/pods/c981b8ca-3c85-4e43-9bfb-d4e2398dec7a/volumes" Nov 03 00:04:06.937979 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:06.937947 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:08.978128 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:08.978078 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" containerID="cri-o://90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5" gracePeriod=25 Nov 03 00:04:09.317217 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:09.317102 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:04:10.710701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.710672 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 03 00:04:10.726015 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.725931 4494 generic.go:356] "Generic (PLEG): container finished" podID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerID="a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754" exitCode=0 Nov 03 00:04:10.726015 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.725989 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" Nov 03 00:04:10.726277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.726028 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" event={"ID":"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0","Type":"ContainerDied","Data":"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754"} Nov 03 00:04:10.726277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.726101 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk" event={"ID":"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0","Type":"ContainerDied","Data":"94193efc083e066423593ec0660a19f5f32bf5a771073f5ae2035d5307576e1e"} Nov 03 00:04:10.726277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.726142 4494 scope.go:122] "RemoveContainer" containerID="a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754" Nov 03 00:04:10.736529 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.736506 4494 scope.go:122] "RemoveContainer" containerID="a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754" Nov 03 00:04:10.736820 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:10.736788 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754\": container with ID starting with a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754 not found: ID does not exist" containerID="a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754" Nov 03 00:04:10.736944 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.736823 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754"} err="failed to get container status \"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754\": rpc error: code = NotFound desc = could not find container \"a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754\": container with ID starting with a76af22a81a993a33be7f960a60cc647f59fa5b8dae2686da523a9c33fae9754 not found: ID does not exist" Nov 03 00:04:10.811981 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.811948 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\") pod \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " Nov 03 00:04:10.812202 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.812044 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\" (UniqueName: \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\") pod \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\" (UID: \"eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0\") " Nov 03 00:04:10.812437 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.812414 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume" pod "eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" (UID: "eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:10.812596 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.812583 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:10.814635 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.814608 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v" pod "eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" (UID: "eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0"). InnerVolumeSpecName "kube-api-access-dlh6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:10.913829 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:10.913790 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh6v\" (UniqueName: \"kubernetes.io/projected/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0-kube-api-access-dlh6v\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:11.019307 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:11.019202 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:11.149689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:11.149654 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 03 00:04:11.152812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:11.152792 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f46b61-6b74b99c99-wwjmk"] Nov 03 00:04:11.694263 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:11.694235 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" path="/var/lib/kubelet/pods/eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0/volumes" Nov 03 00:04:12.281623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:12.281591 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:17.194893 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:17.194858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:17.563971 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:17.563874 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:04:18.765817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:18.765783 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:21.641467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:21.641427 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:22.085773 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:22.085743 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:26.981453 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:26.981411 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:28.765603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:28.765563 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:04:29.259647 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.259615 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 03 00:04:29.372852 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.372754 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\") pod \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " Nov 03 00:04:29.373031 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.373001 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\" (UniqueName: \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\") pod \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\" (UID: \"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df\") " Nov 03 00:04:29.373209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.373154 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume" pod "515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" (UID: "515b5a7c-2f3f-4696-bc30-97d6b9c9d4df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:29.373405 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.373382 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:29.375318 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.375299 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6" pod "515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" (UID: "515b5a7c-2f3f-4696-bc30-97d6b9c9d4df"). InnerVolumeSpecName "kube-api-access-bvks6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:29.475063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.475029 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvks6\" (UniqueName: \"kubernetes.io/projected/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df-kube-api-access-bvks6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:04:29.847457 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.847418 4494 generic.go:356] "Generic (PLEG): container finished" podID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerID="90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5" exitCode=0 Nov 03 00:04:29.847897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.847471 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" Nov 03 00:04:29.847897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.847505 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" event={"ID":"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df","Type":"ContainerDied","Data":"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5"} Nov 03 00:04:29.847897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.847567 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85" event={"ID":"515b5a7c-2f3f-4696-bc30-97d6b9c9d4df","Type":"ContainerDied","Data":"ce4aae2eeb4b61c2810f892457af96df515e2d2f40a2d6761e69fcd224542d20"} Nov 03 00:04:29.847897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.847596 4494 scope.go:122] "RemoveContainer" containerID="90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5" Nov 03 00:04:29.858222 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.858203 4494 scope.go:122] "RemoveContainer" containerID="90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5" Nov 03 00:04:29.858499 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:29.858478 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5\": container with ID starting with 90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5 not found: ID does not exist" containerID="90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5" Nov 03 00:04:29.858545 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:29.858507 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5"} err="failed to get container status \"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5\": rpc error: code = NotFound desc = could not find container \"90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5\": container with ID starting with 90233f1c07a1915b96a6b8447481502775db83c8416f46df3a9bb956501d5bb5 not found: ID does not exist" Nov 03 00:04:30.015906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:30.015821 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 03 00:04:30.066857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:30.066817 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-915ccb-5fbdcf6747-s7r85"] Nov 03 00:04:31.694566 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:31.694536 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" path="/var/lib/kubelet/pods/515b5a7c-2f3f-4696-bc30-97d6b9c9d4df/volumes" Nov 03 00:04:31.802590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:31.802557 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:36.649806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:36.649775 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:39.784114 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:39.784059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:39.928122 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:39.928090 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:41.566235 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:41.566194 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:46.452821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:46.452789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:47.560329 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:04:47.560291 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:04:51.314972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:51.314940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:04:56.245527 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:04:56.245486 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:01.008545 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.008510 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:05:01.009411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009389 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:05:01.009411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009411 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009458 4494 cpu_manager.go:414] "containerMap: removing container" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009465 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009492 4494 cpu_manager.go:414] "containerMap: removing container" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009501 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009515 4494 cpu_manager.go:414] "containerMap: removing container" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009523 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009533 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7d0b840d-1529-48aa-98ae-bff9989f9548" containerName="maas-api" Nov 03 00:05:01.009623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009540 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0b840d-1529-48aa-98ae-bff9989f9548" containerName="maas-api" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009687 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="515b5a7c-2f3f-4696-bc30-97d6b9c9d4df" containerName="payload-pre-processing" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009701 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaec98fb-d3b8-4d32-b5b0-9316fd7f11c0" containerName="payload-pre-processing" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009711 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0b840d-1529-48aa-98ae-bff9989f9548" containerName="maas-api" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009729 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009741 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:05:01.010058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.009755 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f1c709-ac51-46d8-923c-0b92234ca076" containerName="payload-pre-processing" Nov 03 00:05:01.013575 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.013553 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.092190 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.092136 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5ck\" (UniqueName: \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.092381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.092204 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.092381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.092362 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.092476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.092416 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.118095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.118063 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9b6c9e-serving-cert\"" Nov 03 00:05:01.121286 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.121259 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:05:01.192133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.192104 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:01.193381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.193357 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5ck\" (UniqueName: \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.193520 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.193406 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.193586 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.193534 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.193652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.193632 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.193710 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:01.193647 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-serving-cert: secret "maas-api-e2e-shared-b-w1-9b6c9e-serving-cert" not found Nov 03 00:05:01.193710 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:01.193704 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls podName:b2a85a16-3bbf-4607-aaaa-ff32cee2c595 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:01.693685354 +0000 UTC m=+4616.123555207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls") pod "maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" (UID: "b2a85a16-3bbf-4607-aaaa-ff32cee2c595") : secret "maas-api-e2e-shared-b-w1-9b6c9e-serving-cert" not found Nov 03 00:05:01.194235 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.194215 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.196028 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.196007 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.278623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.275634 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5ck\" (UniqueName: \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.358660 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.358621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:01.506345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.506292 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:01.700845 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.700810 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.703411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.703390 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:01.927787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:01.927750 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:02.307877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:02.307844 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:05:02.309540 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:05:02.309510 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a85a16_3bbf_4607_aaaa_ff32cee2c595.slice/crio-ef6e4098b8ad34ae867a539e3b762cc43f7fa9a41f0765175dbb4864cc7c2778 WatchSource:0}: Error finding container ef6e4098b8ad34ae867a539e3b762cc43f7fa9a41f0765175dbb4864cc7c2778: Status 404 returned error can't find the container with id ef6e4098b8ad34ae867a539e3b762cc43f7fa9a41f0765175dbb4864cc7c2778 Nov 03 00:05:03.117030 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:03.116990 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" event={"ID":"b2a85a16-3bbf-4607-aaaa-ff32cee2c595","Type":"ContainerStarted","Data":"ef6e4098b8ad34ae867a539e3b762cc43f7fa9a41f0765175dbb4864cc7c2778"} Nov 03 00:05:04.125182 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:04.125143 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" event={"ID":"b2a85a16-3bbf-4607-aaaa-ff32cee2c595","Type":"ContainerStarted","Data":"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5"} Nov 03 00:05:04.125602 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:04.125349 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:04.246044 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:04.245993 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" podStartSLOduration=3.165462742 podStartE2EDuration="4.245978838s" totalImagesPullingTime="1.080516096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:04.24325246 +0000 UTC m=+4618.673122321" watchObservedRunningTime="2026-11-03 00:05:04.245978838 +0000 UTC m=+4618.675848728" Nov 03 00:05:07.447000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:07.446963 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:08.837251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:08.837220 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:08.991164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:08.991126 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:10.137712 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:10.137640 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:05:10.274002 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:10.273962 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:10.621876 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:10.621833 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 03 00:05:10.622157 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:10.622124 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" containerID="cri-o://1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c" gracePeriod=30 Nov 03 00:05:10.658106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:10.658076 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:11.416133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:11.416081 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:12.616522 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:12.616473 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:12.985010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:12.984976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:15.890124 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:15.890063 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 03 00:05:16.021589 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021546 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021629 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021677 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021749 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021903 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021778 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021903 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021839 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.021986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.021931 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.022057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.022039 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.022321 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.022283 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\") pod \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\" (UID: \"41a81eb1-21f9-4e99-9448-e9c3602b8b16\") " Nov 03 00:05:16.022601 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.022577 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:16.022759 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.022726 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:16.023100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.022919 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:16.023417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.023191 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.023417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.023213 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.023417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.023227 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.023725 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.023687 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:16.024074 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.024049 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:16.025527 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.025498 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "kube-api-access-jbbb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:16.026043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.026017 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:05:16.026557 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.026532 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:16.026657 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.026624 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token" pod "41a81eb1-21f9-4e99-9448-e9c3602b8b16" (UID: "41a81eb1-21f9-4e99-9448-e9c3602b8b16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:16.124459 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124427 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.124459 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124452 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.124459 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124464 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.124706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124473 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbbb9\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-kube-api-access-jbbb9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.124706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124482 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a81eb1-21f9-4e99-9448-e9c3602b8b16-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.124706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.124491 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a81eb1-21f9-4e99-9448-e9c3602b8b16-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:16.200966 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.200932 4494 generic.go:356] "Generic (PLEG): container finished" podID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerID="1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c" exitCode=0 Nov 03 00:05:16.201200 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.200973 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerDied","Data":"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c"} Nov 03 00:05:16.201200 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.201043 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" event={"ID":"41a81eb1-21f9-4e99-9448-e9c3602b8b16","Type":"ContainerDied","Data":"010523e89e646b03c46024a13ce405feb20f61bd3a7d6de34b16c6ef52c482bd"} Nov 03 00:05:16.201200 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.201060 4494 scope.go:122] "RemoveContainer" containerID="1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c" Nov 03 00:05:16.201200 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.200984 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc" Nov 03 00:05:16.218898 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.213647 4494 scope.go:122] "RemoveContainer" containerID="c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2" Nov 03 00:05:16.226343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.226325 4494 scope.go:122] "RemoveContainer" containerID="1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c" Nov 03 00:05:16.226624 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:16.226603 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c\": container with ID starting with 1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c not found: ID does not exist" containerID="1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c" Nov 03 00:05:16.226673 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.226653 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c"} err="failed to get container status \"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c\": rpc error: code = NotFound desc = could not find container \"1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c\": container with ID starting with 1a1ac2f348714c4c67eec1a096dbf9c6a8a991ab2bb672af45e243c7568b242c not found: ID does not exist" Nov 03 00:05:16.226718 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.226673 4494 scope.go:122] "RemoveContainer" containerID="c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2" Nov 03 00:05:16.226949 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:16.226929 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2\": container with ID starting with c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2 not found: ID does not exist" containerID="c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2" Nov 03 00:05:16.227010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.226955 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2"} err="failed to get container status \"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2\": rpc error: code = NotFound desc = could not find container \"c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2\": container with ID starting with c7c0c36dea6b5ca56ce3f70ec894390b6f2e2a0cf0403c943591e7680cd7fbb2 not found: ID does not exist" Nov 03 00:05:16.369144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.369111 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 03 00:05:16.374780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:16.374759 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f46b61-openshift-default-585fcd66c9-kmmrc"] Nov 03 00:05:17.490873 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:17.490842 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:17.563021 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:17.562941 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:05:17.694859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:17.694826 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" path="/var/lib/kubelet/pods/41a81eb1-21f9-4e99-9448-e9c3602b8b16/volumes" Nov 03 00:05:17.719960 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:17.719932 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:18.260086 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:18.260054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:19.115562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:19.115522 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:19.260474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:19.260437 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:19.432467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:19.432435 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:19.781298 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:19.781221 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:21.930297 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:21.930258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:22.024889 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:22.024853 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:05:22.716977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:22.716943 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 03 00:05:22.823043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:22.823005 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 03 00:05:23.105407 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:23.105289 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:23.305314 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:23.305264 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:24.763270 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:24.763225 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:26.156908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:26.156864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:27.273741 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:27.273701 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:27.427931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:27.427896 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:27.718065 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:27.718015 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" containerID="cri-o://ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12" gracePeriod=25 Nov 03 00:05:27.824137 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:27.824087 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" containerID="cri-o://9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e" gracePeriod=25 Nov 03 00:05:28.799568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:28.799534 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.314790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:29.314752 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.732045 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:29.732012 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:29.913787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:29.913753 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:30.419422 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:30.419382 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:05:30.419609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:30.419445 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:05:32.327961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:32.327925 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:33.470548 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.470518 4494 generic.go:356] "Generic (PLEG): container finished" podID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerID="0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb" exitCode=0 Nov 03 00:05:33.470993 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.470588 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerDied","Data":"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb"} Nov 03 00:05:33.471046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.470997 4494 scope.go:122] "RemoveContainer" containerID="0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb" Nov 03 00:05:33.472717 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.472686 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:05:33.472819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.472755 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:05:33.472819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.472786 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:05:33.483066 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.483031 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:33.483066 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:33.483060 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:34.478664 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:34.478626 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerStarted","Data":"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f"} Nov 03 00:05:34.765875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:34.765796 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:35.175473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:35.175439 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:35.359613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:35.359581 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:05:35.360015 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:35.359907 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 03 00:05:35.360145 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:35.360037 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 03 00:05:36.355639 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:36.355602 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 03 00:05:36.356060 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:36.355657 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 03 00:05:36.686044 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:36.686012 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:37.355670 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:37.355632 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 03 00:05:37.356297 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:37.356257 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 03 00:05:38.355617 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:38.355582 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 03 00:05:38.355841 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:38.355648 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 03 00:05:39.355649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:39.355597 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Nov 03 00:05:39.355884 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:39.355665 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Nov 03 00:05:40.155022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:40.154989 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:40.358710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:40.358673 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:05:40.359097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:40.358985 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:05:40.359723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:40.359702 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:05:42.606796 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:42.606745 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:42.776324 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:42.776280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:43.403731 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:43.403690 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:43.483571 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:43.483534 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:43.483743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:43.483534 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:05:45.231557 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:45.231522 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:47.560218 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:47.560165 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:05:48.003416 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.003379 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 03 00:05:48.046324 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.046291 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\" (UniqueName: \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\") pod \"39fd866e-2b8c-460b-b254-6fdf8f49a217\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " Nov 03 00:05:48.046528 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.046393 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\") pod \"39fd866e-2b8c-460b-b254-6fdf8f49a217\" (UID: \"39fd866e-2b8c-460b-b254-6fdf8f49a217\") " Nov 03 00:05:48.046867 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.046837 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume" pod "39fd866e-2b8c-460b-b254-6fdf8f49a217" (UID: "39fd866e-2b8c-460b-b254-6fdf8f49a217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:48.047208 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.047146 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39fd866e-2b8c-460b-b254-6fdf8f49a217-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:48.049610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.049586 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk" pod "39fd866e-2b8c-460b-b254-6fdf8f49a217" (UID: "39fd866e-2b8c-460b-b254-6fdf8f49a217"). InnerVolumeSpecName "kube-api-access-h8kgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:48.084784 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.084712 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 03 00:05:48.148301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.148266 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\" (UniqueName: \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\") pod \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " Nov 03 00:05:48.148486 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.148457 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\") pod \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\" (UID: \"ec6874d4-1de4-4de2-9aae-f4cab811cac2\") " Nov 03 00:05:48.148878 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.148852 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume" pod "ec6874d4-1de4-4de2-9aae-f4cab811cac2" (UID: "ec6874d4-1de4-4de2-9aae-f4cab811cac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:48.149018 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.149004 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kgk\" (UniqueName: \"kubernetes.io/projected/39fd866e-2b8c-460b-b254-6fdf8f49a217-kube-api-access-h8kgk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:48.149075 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.149025 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6874d4-1de4-4de2-9aae-f4cab811cac2-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:48.150520 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.150500 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7" pod "ec6874d4-1de4-4de2-9aae-f4cab811cac2" (UID: "ec6874d4-1de4-4de2-9aae-f4cab811cac2"). InnerVolumeSpecName "kube-api-access-v9nv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:48.250111 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.250079 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nv7\" (UniqueName: \"kubernetes.io/projected/ec6874d4-1de4-4de2-9aae-f4cab811cac2-kube-api-access-v9nv7\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:48.571084 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.571040 4494 generic.go:356] "Generic (PLEG): container finished" podID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerID="ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12" exitCode=0 Nov 03 00:05:48.571636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.571097 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" Nov 03 00:05:48.571636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.571098 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" event={"ID":"39fd866e-2b8c-460b-b254-6fdf8f49a217","Type":"ContainerDied","Data":"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12"} Nov 03 00:05:48.571636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.571189 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj" event={"ID":"39fd866e-2b8c-460b-b254-6fdf8f49a217","Type":"ContainerDied","Data":"070eefac25f299bcf03868e797a0474ae8aa08a2db6f375522bb02dbde242346"} Nov 03 00:05:48.571636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.571224 4494 scope.go:122] "RemoveContainer" containerID="ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12" Nov 03 00:05:48.572667 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.572636 4494 generic.go:356] "Generic (PLEG): container finished" podID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerID="9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e" exitCode=0 Nov 03 00:05:48.572813 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.572692 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" Nov 03 00:05:48.572813 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.572731 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" event={"ID":"ec6874d4-1de4-4de2-9aae-f4cab811cac2","Type":"ContainerDied","Data":"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e"} Nov 03 00:05:48.572813 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.572776 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp" event={"ID":"ec6874d4-1de4-4de2-9aae-f4cab811cac2","Type":"ContainerDied","Data":"dbe56bf4b06553f3a69281521f0ac1a88ae90cb9876a46b09227bcde9641c416"} Nov 03 00:05:48.583560 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.583442 4494 scope.go:122] "RemoveContainer" containerID="ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12" Nov 03 00:05:48.583919 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:48.583787 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12\": container with ID starting with ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12 not found: ID does not exist" containerID="ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12" Nov 03 00:05:48.583919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.583813 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12"} err="failed to get container status \"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12\": rpc error: code = NotFound desc = could not find container \"ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12\": container with ID starting with ba066b0c315bf28f60783746eb5d56329ce78c2863d8fbda3b485f705fe14e12 not found: ID does not exist" Nov 03 00:05:48.583919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.583831 4494 scope.go:122] "RemoveContainer" containerID="9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e" Nov 03 00:05:48.593661 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.593644 4494 scope.go:122] "RemoveContainer" containerID="9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e" Nov 03 00:05:48.593933 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:05:48.593907 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e\": container with ID starting with 9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e not found: ID does not exist" containerID="9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e" Nov 03 00:05:48.593976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.593942 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e"} err="failed to get container status \"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e\": rpc error: code = NotFound desc = could not find container \"9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e\": container with ID starting with 9c39353ba8e321f3efba937c6159c2d546b9891c1d3bb3d038ce9f6d2056657e not found: ID does not exist" Nov 03 00:05:48.738345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.738317 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 03 00:05:48.787133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.787104 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1f0009-6d85847c55-55xfp"] Nov 03 00:05:48.851766 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.851665 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 03 00:05:48.855628 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:48.855600 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1f0009-554d6d9d9-2czgj"] Nov 03 00:05:49.088881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:49.088848 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:49.693797 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:49.693764 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" path="/var/lib/kubelet/pods/39fd866e-2b8c-460b-b254-6fdf8f49a217/volumes" Nov 03 00:05:49.694246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:49.694230 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" path="/var/lib/kubelet/pods/ec6874d4-1de4-4de2-9aae-f4cab811cac2/volumes" Nov 03 00:05:50.247272 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:50.247232 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:53.891928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:53.891893 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:55.172217 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:55.172150 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:55.315040 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:55.315006 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:55.458942 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:55.458835 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:05:57.781306 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:05:57.781273 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:06:00.078140 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:00.078091 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:04.963058 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:04.963018 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:05.629826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:05.629790 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:05.772806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:05.772773 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:09.811282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:09.811243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:12.343811 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:12.343773 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:12.503659 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:12.503625 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:12.803978 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:12.803936 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:14.741358 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:14.741322 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:17.560806 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:17.560771 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:06:18.484935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:18.484858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:19.669998 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:19.669966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:22.268299 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:22.268270 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:24.179585 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.179555 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 03 00:06:24.179990 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.179795 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" podUID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" containerName="maas-api" containerID="cri-o://455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891" gracePeriod=30 Nov 03 00:06:24.282551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.282516 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:24.477062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.477034 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 03 00:06:24.581196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.568693 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\") pod \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " Nov 03 00:06:24.581196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.568815 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\" (UniqueName: \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\") pod \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " Nov 03 00:06:24.581196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.568893 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\") pod \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " Nov 03 00:06:24.581196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.568956 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") pod \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\" (UID: \"5641f8cb-eb90-4227-b066-f9b7bc1f5581\") " Nov 03 00:06:24.581196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.579385 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca" pod "5641f8cb-eb90-4227-b066-f9b7bc1f5581" (UID: "5641f8cb-eb90-4227-b066-f9b7bc1f5581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:24.610790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.609814 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls" pod "5641f8cb-eb90-4227-b066-f9b7bc1f5581" (UID: "5641f8cb-eb90-4227-b066-f9b7bc1f5581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:24.615195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.613390 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn" pod "5641f8cb-eb90-4227-b066-f9b7bc1f5581" (UID: "5641f8cb-eb90-4227-b066-f9b7bc1f5581"). InnerVolumeSpecName "kube-api-access-7prcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:24.646246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.646153 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs" pod "5641f8cb-eb90-4227-b066-f9b7bc1f5581" (UID: "5641f8cb-eb90-4227-b066-f9b7bc1f5581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:24.671670 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.671633 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5641f8cb-eb90-4227-b066-f9b7bc1f5581-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:24.671670 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.671672 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prcn\" (UniqueName: \"kubernetes.io/projected/5641f8cb-eb90-4227-b066-f9b7bc1f5581-kube-api-access-7prcn\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:24.671923 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.671692 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:24.671923 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.671712 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5641f8cb-eb90-4227-b066-f9b7bc1f5581-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:24.823856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.823772 4494 generic.go:356] "Generic (PLEG): container finished" podID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" containerID="455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891" exitCode=0 Nov 03 00:06:24.823856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.823815 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" Nov 03 00:06:24.823856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.823847 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" event={"ID":"5641f8cb-eb90-4227-b066-f9b7bc1f5581","Type":"ContainerDied","Data":"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891"} Nov 03 00:06:24.824209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.823902 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82" event={"ID":"5641f8cb-eb90-4227-b066-f9b7bc1f5581","Type":"ContainerDied","Data":"79058b8836e02898d4b6235e28507903806971d5c10e8daee803f1efc3460497"} Nov 03 00:06:24.824209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.823930 4494 scope.go:122] "RemoveContainer" containerID="455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891" Nov 03 00:06:24.828343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.828313 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:24.834543 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.834526 4494 scope.go:122] "RemoveContainer" containerID="455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891" Nov 03 00:06:24.834801 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:24.834782 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891\": container with ID starting with 455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891 not found: ID does not exist" containerID="455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891" Nov 03 00:06:24.834879 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:24.834812 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891"} err="failed to get container status \"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891\": rpc error: code = NotFound desc = could not find container \"455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891\": container with ID starting with 455283c37e56f1eb0fa9659191359e98692de5e7453df35f723f5d8c61bda891 not found: ID does not exist" Nov 03 00:06:25.001538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:25.001510 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 03 00:06:25.004014 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:25.003992 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1f0009-74d8b89b66-6dm82"] Nov 03 00:06:25.695386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:25.695358 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" path="/var/lib/kubelet/pods/5641f8cb-eb90-4227-b066-f9b7bc1f5581/volumes" Nov 03 00:06:26.060387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:26.060294 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:30.963627 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:30.963595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:33.137191 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:33.137132 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:33.280768 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:33.280733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:34.725046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:34.725007 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:35.187270 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:35.187231 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:35.191831 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:35.191805 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 03 00:06:35.192036 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:35.192000 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" containerID="cri-o://06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4" gracePeriod=30 Nov 03 00:06:35.639806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:35.639712 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:36.148446 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:36.148408 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:40.473003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.472940 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 03 00:06:40.558278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558239 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558344 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558385 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558419 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558454 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558472 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558558 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558624 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558658 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:40.558809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558691 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\") pod \"82d840d5-3a79-4ef6-8214-0bce2361db7b\" (UID: \"82d840d5-3a79-4ef6-8214-0bce2361db7b\") " Nov 03 00:06:40.558809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558745 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:40.559296 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.558884 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:40.559296 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559036 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:40.559296 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559124 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:40.559496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559335 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.559496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559354 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.559496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559370 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.559496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559383 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.559496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.559395 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d840d5-3a79-4ef6-8214-0bce2361db7b-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.560708 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.560684 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:40.560817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.560764 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:40.560919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.560903 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "kube-api-access-rz9rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:40.560970 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.560946 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo" pod "82d840d5-3a79-4ef6-8214-0bce2361db7b" (UID: "82d840d5-3a79-4ef6-8214-0bce2361db7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:40.599120 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.599091 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 03 00:06:40.613473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.613442 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 03 00:06:40.660653 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.660617 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.660653 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.660649 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.660877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.660661 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9rc\" (UniqueName: \"kubernetes.io/projected/82d840d5-3a79-4ef6-8214-0bce2361db7b-kube-api-access-rz9rc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.660877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.660670 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d840d5-3a79-4ef6-8214-0bce2361db7b-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:40.693761 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.693726 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:40.840766 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.840683 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:40.945111 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.945079 4494 generic.go:356] "Generic (PLEG): container finished" podID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerID="06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4" exitCode=0 Nov 03 00:06:40.945335 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.945132 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" Nov 03 00:06:40.945335 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.945162 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerDied","Data":"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4"} Nov 03 00:06:40.945335 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.945259 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h" event={"ID":"82d840d5-3a79-4ef6-8214-0bce2361db7b","Type":"ContainerDied","Data":"f131b681f100487294a4b5f78d8af9ebd9908d7f7976b0518509b5e2842ef754"} Nov 03 00:06:40.945335 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.945289 4494 scope.go:122] "RemoveContainer" containerID="06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4" Nov 03 00:06:40.996197 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:40.995758 4494 scope.go:122] "RemoveContainer" containerID="59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965" Nov 03 00:06:41.031592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.031572 4494 scope.go:122] "RemoveContainer" containerID="06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4" Nov 03 00:06:41.031998 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:41.031977 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4\": container with ID starting with 06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4 not found: ID does not exist" containerID="06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4" Nov 03 00:06:41.032099 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.032010 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4"} err="failed to get container status \"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4\": rpc error: code = NotFound desc = could not find container \"06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4\": container with ID starting with 06c25f841a99c97419a49331605b643ef9a74c36342e5247314bc969ebf953c4 not found: ID does not exist" Nov 03 00:06:41.032099 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.032034 4494 scope.go:122] "RemoveContainer" containerID="59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965" Nov 03 00:06:41.032376 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:41.032357 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965\": container with ID starting with 59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965 not found: ID does not exist" containerID="59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965" Nov 03 00:06:41.032445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.032387 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965"} err="failed to get container status \"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965\": rpc error: code = NotFound desc = could not find container \"59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965\": container with ID starting with 59bd21e7d796431f8884aeb70603f2c40ab08330b633cb7770afc49c5dcdc965 not found: ID does not exist" Nov 03 00:06:41.123687 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.123604 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 03 00:06:41.132110 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.132077 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1f0009-openshift-default-77c6d4f6b7-dwp2h"] Nov 03 00:06:41.694343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:41.694300 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" path="/var/lib/kubelet/pods/82d840d5-3a79-4ef6-8214-0bce2361db7b/volumes" Nov 03 00:06:42.444674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:42.444642 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:46.882972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:46.882932 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:47.039233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:47.039184 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:47.373668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:47.373636 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:47.563541 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:47.563326 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:06:48.503643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:48.503610 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:48.979098 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:48.979066 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:48.984287 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:48.984264 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 03 00:06:48.984517 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:48.984491 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" containerID="cri-o://2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283" gracePeriod=30 Nov 03 00:06:49.549821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:49.549789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:50.485198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:50.485137 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:52.554850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:52.554815 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:52.647214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:52.647155 4494 patch_prober.go:29] interesting pod/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:52.647454 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:52.647258 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:52.704977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:52.704934 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:53.581533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:53.581497 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:54.191710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.183141 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 03 00:06:54.215085 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.215042 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:54.269017 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.268988 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 03 00:06:54.346998 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.346967 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347006 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347074 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347107 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347159 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347221 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347292 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347366 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.347564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347417 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.347564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347481 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.347564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347519 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\") pod \"61a87125-f959-4252-8e96-ede7e8de6a14\" (UID: \"61a87125-f959-4252-8e96-ede7e8de6a14\") " Nov 03 00:06:54.348012 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347656 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.348012 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:54.348012 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.347686 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.348210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.348023 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.348210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.348051 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a87125-f959-4252-8e96-ede7e8de6a14-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.348210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.348064 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.348210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.348077 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.348210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.348090 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.349525 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.349496 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:54.349675 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.349542 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:54.349893 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.349870 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "kube-api-access-qtmsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:54.349971 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.349944 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token" pod "61a87125-f959-4252-8e96-ede7e8de6a14" (UID: "61a87125-f959-4252-8e96-ede7e8de6a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:54.366059 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.366035 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:54.448952 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.448920 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.448952 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.448948 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a87125-f959-4252-8e96-ede7e8de6a14-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.449155 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.448963 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a87125-f959-4252-8e96-ede7e8de6a14-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.449155 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.448976 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmsg\" (UniqueName: \"kubernetes.io/projected/61a87125-f959-4252-8e96-ede7e8de6a14-kube-api-access-qtmsg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:54.522657 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.522524 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:54.667644 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:54.667601 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:55.051343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.051311 4494 generic.go:356] "Generic (PLEG): container finished" podID="61a87125-f959-4252-8e96-ede7e8de6a14" containerID="2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283" exitCode=0 Nov 03 00:06:55.051533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.051362 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" Nov 03 00:06:55.051533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.051399 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerDied","Data":"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283"} Nov 03 00:06:55.051533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.051457 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn" event={"ID":"61a87125-f959-4252-8e96-ede7e8de6a14","Type":"ContainerDied","Data":"398ffb0db3813b6645ec4985b1fa83bb165267233dc7a7689dade61dc3150fcb"} Nov 03 00:06:55.051533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.051480 4494 scope.go:122] "RemoveContainer" containerID="2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283" Nov 03 00:06:55.063480 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.063461 4494 scope.go:122] "RemoveContainer" containerID="f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948" Nov 03 00:06:55.072772 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.072755 4494 scope.go:122] "RemoveContainer" containerID="2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283" Nov 03 00:06:55.073048 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:55.073030 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283\": container with ID starting with 2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283 not found: ID does not exist" containerID="2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283" Nov 03 00:06:55.073114 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.073058 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283"} err="failed to get container status \"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283\": rpc error: code = NotFound desc = could not find container \"2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283\": container with ID starting with 2c79e152c08a29adadb744ba7ce2a0f13ca95b04ab387556b286c0fc9efbe283 not found: ID does not exist" Nov 03 00:06:55.073114 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.073075 4494 scope.go:122] "RemoveContainer" containerID="f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948" Nov 03 00:06:55.073336 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:55.073320 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948\": container with ID starting with f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948 not found: ID does not exist" containerID="f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948" Nov 03 00:06:55.073384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.073340 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948"} err="failed to get container status \"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948\": rpc error: code = NotFound desc = could not find container \"f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948\": container with ID starting with f8a583cf6f546cbf78966fe65b938541b5db7047d1143204ea11cd148c1c8948 not found: ID does not exist" Nov 03 00:06:55.223318 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.223279 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 03 00:06:55.227610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.227528 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ec2f9a-openshift-default-6cf98f459c-892qn"] Nov 03 00:06:55.620977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.620935 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" containerID="cri-o://5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e" gracePeriod=45 Nov 03 00:06:55.639696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.639659 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" containerID="cri-o://dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58" gracePeriod=45 Nov 03 00:06:55.714910 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.714875 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" path="/var/lib/kubelet/pods/61a87125-f959-4252-8e96-ede7e8de6a14/volumes" Nov 03 00:06:55.974051 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.974020 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 03 00:06:55.997316 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:55.997289 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 03 00:06:56.060606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.060567 4494 generic.go:356] "Generic (PLEG): container finished" podID="243999e3-7f65-4446-930d-b4eb8d180be0" containerID="5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e" exitCode=0 Nov 03 00:06:56.060780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.060618 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" Nov 03 00:06:56.060780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.060661 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" event={"ID":"243999e3-7f65-4446-930d-b4eb8d180be0","Type":"ContainerDied","Data":"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e"} Nov 03 00:06:56.060780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.060715 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" event={"ID":"243999e3-7f65-4446-930d-b4eb8d180be0","Type":"ContainerDied","Data":"43d7238b3a4acb1556446464596c3ea2b48100707d24afec6334ac61cc46a1a1"} Nov 03 00:06:56.060780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.060734 4494 scope.go:122] "RemoveContainer" containerID="5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e" Nov 03 00:06:56.063159 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.063137 4494 generic.go:356] "Generic (PLEG): container finished" podID="fab4b6c4-d139-4420-993c-120f78771e0b" containerID="dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58" exitCode=0 Nov 03 00:06:56.063318 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.063213 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" event={"ID":"fab4b6c4-d139-4420-993c-120f78771e0b","Type":"ContainerDied","Data":"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58"} Nov 03 00:06:56.063318 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.063252 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" Nov 03 00:06:56.063318 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.063265 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc" event={"ID":"fab4b6c4-d139-4420-993c-120f78771e0b","Type":"ContainerDied","Data":"c170d7fc2bf8a1e48db2b5a6676c755641303996988617aabee2374635dcfddc"} Nov 03 00:06:56.071687 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.071670 4494 scope.go:122] "RemoveContainer" containerID="5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e" Nov 03 00:06:56.072036 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:56.072010 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e\": container with ID starting with 5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e not found: ID does not exist" containerID="5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e" Nov 03 00:06:56.072233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.072039 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e"} err="failed to get container status \"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e\": rpc error: code = NotFound desc = could not find container \"5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e\": container with ID starting with 5df85cbe2344bfe6406239134ad0da667c626c053feac1ddb6879cc82226b36e not found: ID does not exist" Nov 03 00:06:56.072233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.072056 4494 scope.go:122] "RemoveContainer" containerID="dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58" Nov 03 00:06:56.080964 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.080946 4494 scope.go:122] "RemoveContainer" containerID="dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58" Nov 03 00:06:56.081266 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:06:56.081246 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58\": container with ID starting with dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58 not found: ID does not exist" containerID="dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58" Nov 03 00:06:56.081379 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.081271 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58"} err="failed to get container status \"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58\": rpc error: code = NotFound desc = could not find container \"dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58\": container with ID starting with dc6f5fe76f41ad6092dd68c4f1e4bd88a8dad025b64f7f3f80202a1bca0d3e58 not found: ID does not exist" Nov 03 00:06:56.110649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110613 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.110822 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110726 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.110822 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110776 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.110822 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110811 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.110934 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110885 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\" (UniqueName: \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.110934 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110926 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.111004 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110966 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.111004 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.110995 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.111103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.111087 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\" (UniqueName: \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.111152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.111138 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\" (UniqueName: \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.111220 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.111206 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\") pod \"243999e3-7f65-4446-930d-b4eb8d180be0\" (UID: \"243999e3-7f65-4446-930d-b4eb8d180be0\") " Nov 03 00:06:56.111276 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.111262 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\") pod \"fab4b6c4-d139-4420-993c-120f78771e0b\" (UID: \"fab4b6c4-d139-4420-993c-120f78771e0b\") " Nov 03 00:06:56.112285 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112202 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.112420 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112320 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.112538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112513 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.112606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112533 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.112606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112582 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.112810 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.112789 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.115339 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.115310 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "kube-api-access-vhdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:56.116645 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.116618 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:56.116645 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.116628 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.116645 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.116634 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "kube-api-access-6j55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:56.116897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.116654 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs" pod "243999e3-7f65-4446-930d-b4eb8d180be0" (UID: "243999e3-7f65-4446-930d-b4eb8d180be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:56.116897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.116689 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm" pod "fab4b6c4-d139-4420-993c-120f78771e0b" (UID: "fab4b6c4-d139-4420-993c-120f78771e0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:56.213444 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213411 4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243999e3-7f65-4446-930d-b4eb8d180be0-tls-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213444 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213444 4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-tmp-dir\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213456 4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-dshm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213465 4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-model-cache\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213476 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdln\" (UniqueName: \"kubernetes.io/projected/243999e3-7f65-4446-930d-b4eb8d180be0-kube-api-access-vhdln\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213486 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j55t\" (UniqueName: \"kubernetes.io/projected/fab4b6c4-d139-4420-993c-120f78771e0b-kube-api-access-6j55t\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213494 4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-dshm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213502 4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-tmp-dir\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213510 4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4b6c4-d139-4420-993c-120f78771e0b-home\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213517 4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-model-cache\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213526 4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243999e3-7f65-4446-930d-b4eb8d180be0-home\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.213651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.213533 4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4b6c4-d139-4420-993c-120f78771e0b-tls-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:56.486105 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.486025 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 03 00:06:56.491143 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.491116 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7"] Nov 03 00:06:56.601144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.601115 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 03 00:06:56.607441 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.607414 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-96031a/premium-w3-96031a-kserve-fb668585f-hmdxc"] Nov 03 00:06:56.797695 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.797595 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:56.927440 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:56.927392 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-96031a/distinct-2-w3-96031a-kserve-8b66966b9-ndwr7" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:06:57.049418 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:57.049329 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:57.695701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:57.695660 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" path="/var/lib/kubelet/pods/243999e3-7f65-4446-930d-b4eb8d180be0/volumes" Nov 03 00:06:57.696183 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:57.696150 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" path="/var/lib/kubelet/pods/fab4b6c4-d139-4420-993c-120f78771e0b/volumes" Nov 03 00:06:57.740943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:57.740899 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:06:59.184333 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:59.184284 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" containerID="cri-o://b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e" gracePeriod=25 Nov 03 00:06:59.524100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:06:59.524021 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:00.598014 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:00.597931 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:01.195899 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:01.195472 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:07:01.451494 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:01.451411 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:01.595763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:01.595729 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:02.851644 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:02.851612 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:05.456749 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:05.456705 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:05.839746 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:05.839652 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:07:06.196603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.196528 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" podUID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" containerName="payload-pre-processing" containerID="cri-o://d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f" gracePeriod=25 Nov 03 00:07:06.480259 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.480232 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:07:06.559497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.559470 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\" (UniqueName: \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\") pod \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " Nov 03 00:07:06.559696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.559529 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\") pod \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\" (UID: \"5f3b9acd-bad8-4f16-8c14-91286a4f3809\") " Nov 03 00:07:06.559928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.559900 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume" pod "5f3b9acd-bad8-4f16-8c14-91286a4f3809" (UID: "5f3b9acd-bad8-4f16-8c14-91286a4f3809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:06.561816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.561796 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6" pod "5f3b9acd-bad8-4f16-8c14-91286a4f3809" (UID: "5f3b9acd-bad8-4f16-8c14-91286a4f3809"). InnerVolumeSpecName "kube-api-access-7pwc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:06.664675 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.663749 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwc6\" (UniqueName: \"kubernetes.io/projected/5f3b9acd-bad8-4f16-8c14-91286a4f3809-kube-api-access-7pwc6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:06.664875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:06.664683 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3b9acd-bad8-4f16-8c14-91286a4f3809-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:07.139881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.139850 4494 generic.go:356] "Generic (PLEG): container finished" podID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" containerID="d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f" exitCode=0 Nov 03 00:07:07.140195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.139898 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" event={"ID":"5f3b9acd-bad8-4f16-8c14-91286a4f3809","Type":"ContainerDied","Data":"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f"} Nov 03 00:07:07.140195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.139915 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" Nov 03 00:07:07.140195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.139939 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr" event={"ID":"5f3b9acd-bad8-4f16-8c14-91286a4f3809","Type":"ContainerDied","Data":"09b9b319b4d42b3eb3fc2ba295d4abbc0850c1936688c1994e8aabb47e3d6d65"} Nov 03 00:07:07.140195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.139961 4494 scope.go:122] "RemoveContainer" containerID="d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f" Nov 03 00:07:07.150110 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.150092 4494 scope.go:122] "RemoveContainer" containerID="d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f" Nov 03 00:07:07.150387 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:07:07.150365 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f\": container with ID starting with d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f not found: ID does not exist" containerID="d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f" Nov 03 00:07:07.150466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.150396 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f"} err="failed to get container status \"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f\": rpc error: code = NotFound desc = could not find container \"d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f\": container with ID starting with d0a02c92b72e4d593b32dc56cc266c3b5c6530cf9212ca3f6af2d5509966a94f not found: ID does not exist" Nov 03 00:07:07.312524 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.312493 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:07:07.362761 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.362730 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04713b77-66789489bc-9x9lr"] Nov 03 00:07:07.695561 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.695528 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" path="/var/lib/kubelet/pods/5f3b9acd-bad8-4f16-8c14-91286a4f3809/volumes" Nov 03 00:07:07.750553 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:07.750520 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:10.597309 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:10.597270 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:12.345581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:12.345549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:12.667836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:12.667793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:17.562387 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:07:17.562345 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:07:17.637849 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:17.637765 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:18.204091 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:18.204013 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:18.351014 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:18.350984 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:18.827537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:18.827508 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:19.433370 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.433346 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 03 00:07:19.525290 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.525215 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\" (UniqueName: \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\") pod \"66492f3c-f750-466b-b051-74adddec8220\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " Nov 03 00:07:19.525290 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.525258 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\") pod \"66492f3c-f750-466b-b051-74adddec8220\" (UID: \"66492f3c-f750-466b-b051-74adddec8220\") " Nov 03 00:07:19.525727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.525704 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume" pod "66492f3c-f750-466b-b051-74adddec8220" (UID: "66492f3c-f750-466b-b051-74adddec8220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:19.525830 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.525816 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66492f3c-f750-466b-b051-74adddec8220-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:19.527591 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.527568 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s" pod "66492f3c-f750-466b-b051-74adddec8220" (UID: "66492f3c-f750-466b-b051-74adddec8220"). InnerVolumeSpecName "kube-api-access-fnm2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:19.627600 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.627567 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm2s\" (UniqueName: \"kubernetes.io/projected/66492f3c-f750-466b-b051-74adddec8220-kube-api-access-fnm2s\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:07:19.976487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:19.976447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:20.117943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.117904 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:20.229569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.229491 4494 generic.go:356] "Generic (PLEG): container finished" podID="66492f3c-f750-466b-b051-74adddec8220" containerID="b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e" exitCode=0 Nov 03 00:07:20.229569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.229551 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" Nov 03 00:07:20.229756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.229576 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" event={"ID":"66492f3c-f750-466b-b051-74adddec8220","Type":"ContainerDied","Data":"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e"} Nov 03 00:07:20.229756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.229646 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz" event={"ID":"66492f3c-f750-466b-b051-74adddec8220","Type":"ContainerDied","Data":"f583a3b18234657d58f50628946bd4b141796b0087c745bf15c9467879ec0c3c"} Nov 03 00:07:20.229756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.229668 4494 scope.go:122] "RemoveContainer" containerID="b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e" Nov 03 00:07:20.239137 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.239112 4494 scope.go:122] "RemoveContainer" containerID="b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e" Nov 03 00:07:20.239449 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:07:20.239432 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e\": container with ID starting with b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e not found: ID does not exist" containerID="b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e" Nov 03 00:07:20.239514 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.239457 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e"} err="failed to get container status \"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e\": rpc error: code = NotFound desc = could not find container \"b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e\": container with ID starting with b1a8631693269de7cc33f50b8fc5b27e2daa75a7b24847ce0a9d5376e478a88e not found: ID does not exist" Nov 03 00:07:20.395350 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.395318 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 03 00:07:20.446231 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:20.446195 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-96031a-7fb57bcc9-c64nz"] Nov 03 00:07:21.693738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:21.693708 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66492f3c-f750-466b-b051-74adddec8220" path="/var/lib/kubelet/pods/66492f3c-f750-466b-b051-74adddec8220/volumes" Nov 03 00:07:21.875024 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:21.874995 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:22.749760 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:22.748812 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:24.400711 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:24.400678 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:24.584581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:24.584540 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.344188 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345612 4494 cpu_manager.go:414] "containerMap: removing container" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345636 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345666 4494 cpu_manager.go:414] "containerMap: removing container" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345675 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345703 4494 cpu_manager.go:414] "containerMap: removing container" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345712 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345765 4494 cpu_manager.go:414] "containerMap: removing container" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345774 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345793 4494 cpu_manager.go:414] "containerMap: removing container" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345802 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345826 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345833 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345852 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345860 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c981b8ca-3c85-4e43-9bfb-d4e2398dec7a" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345870 4494 cpu_manager.go:414] "containerMap: removing container" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345877 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345900 4494 cpu_manager.go:414] "containerMap: removing container" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345907 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345926 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" containerName="maas-api" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345933 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" containerName="maas-api" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345951 4494 cpu_manager.go:414] "containerMap: removing container" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.345959 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346022 4494 cpu_manager.go:414] "containerMap: removing container" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346030 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346087 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346097 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346119 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346127 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346431 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346461 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6874d4-1de4-4de2-9aae-f4cab811cac2" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346471 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3b9acd-bad8-4f16-8c14-91286a4f3809" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346494 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fd866e-2b8c-460b-b254-6fdf8f49a217" containerName="payload-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346504 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab4b6c4-d139-4420-993c-120f78771e0b" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346521 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="66492f3c-f750-466b-b051-74adddec8220" containerName="payload-pre-processing" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346533 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="243999e3-7f65-4446-930d-b4eb8d180be0" containerName="main" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346541 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346558 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346574 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a87125-f959-4252-8e96-ede7e8de6a14" containerName="istio-proxy" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346584 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5641f8cb-eb90-4227-b066-f9b7bc1f5581" containerName="maas-api" Nov 03 00:07:26.347205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.346593 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a81eb1-21f9-4e99-9448-e9c3602b8b16" containerName="istio-proxy" Nov 03 00:07:26.356057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.356033 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.409501 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.409478 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-96031a-openshift-default-dockercfg-fx8gh\"" Nov 03 00:07:26.420064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420037 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420203 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420073 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420203 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420098 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420255 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420304 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420454 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420332 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420526 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420509 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420576 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.420688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.420611 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.465274 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.465240 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:07:26.521483 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521450 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521499 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521518 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521544 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521567 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521595 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521627 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521672 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.521696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.521699 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.522140 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.522035 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.522140 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.522092 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.522325 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.522188 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.522325 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.522280 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.522496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.522321 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.524317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.524293 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.524508 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.524491 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.583640 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.583602 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.583817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.583692 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\") pod \"e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:26.669797 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:26.669760 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:27.112525 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.112493 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:07:27.113907 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:07:27.113880 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8414450_c908_4418_8983_2f2f9efe1695.slice/crio-f934fd20a296784099d63c187d2834badb737cfa161b78aae2527dd5f6053a60 WatchSource:0}: Error finding container f934fd20a296784099d63c187d2834badb737cfa161b78aae2527dd5f6053a60: Status 404 returned error can't find the container with id f934fd20a296784099d63c187d2834badb737cfa161b78aae2527dd5f6053a60 Nov 03 00:07:27.116445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.116412 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.116515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.116482 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.116558 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.116517 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:07:27.274909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.274879 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" event={"ID":"f8414450-c908-4418-8983-2f2f9efe1695","Type":"ContainerStarted","Data":"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026"} Nov 03 00:07:27.275082 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.274929 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" event={"ID":"f8414450-c908-4418-8983-2f2f9efe1695","Type":"ContainerStarted","Data":"f934fd20a296784099d63c187d2834badb737cfa161b78aae2527dd5f6053a60"} Nov 03 00:07:27.408855 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.408750 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podStartSLOduration=1.408735687 podStartE2EDuration="1.408735687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:07:27.408151309 +0000 UTC m=+4761.838021172" watchObservedRunningTime="2026-11-03 00:07:27.408735687 +0000 UTC m=+4761.838605547" Nov 03 00:07:27.670531 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:27.670494 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:28.547089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:28.547026 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:28.671735 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:28.671697 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 00:07:28.671943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:28.671744 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 00:07:29.670681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:29.670643 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 00:07:29.671136 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:29.670713 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 00:07:29.779762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:29.779727 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:29.923588 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:29.923457 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:30.670147 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:30.670111 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 00:07:30.670363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:30.670203 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 00:07:31.306976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:31.306922 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:31.670800 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:31.670770 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 00:07:31.671000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:31.670820 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 00:07:31.814554 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:31.814510 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:32.582902 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:32.582864 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:32.670514 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:32.670479 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" start-of-body= Nov 03 00:07:32.670702 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:32.670533 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.42:15021/healthz/ready\": dial tcp 10.133.1.42:15021: connect: connection refused" Nov 03 00:07:33.674489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:33.674453 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:33.674953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:33.674819 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:33.675467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:33.675444 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:07:33.797766 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:33.797733 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:38.741025 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:38.740993 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:38.985352 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:38.985303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:39.916612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:39.916577 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:42.049069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:42.049034 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 03 00:07:42.161443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:42.161383 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 03 00:07:42.410516 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:42.410487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:42.560464 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:42.560417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:44.171837 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:44.171794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:47.049857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:47.049813 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" containerID="cri-o://67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592" gracePeriod=25 Nov 03 00:07:47.162069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:47.162023 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" containerID="cri-o://515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88" gracePeriod=25 Nov 03 00:07:47.559889 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:07:47.559851 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:07:48.555470 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:48.555437 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:49.564655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:49.564621 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:49.711930 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:49.711888 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:50.222453 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:50.222411 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:50.222653 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:50.222537 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:07:50.342347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:50.342291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:52.108618 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:52.108586 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:07:53.633146 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:53.633113 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:55.245397 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:55.245353 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:56.826598 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:56.826568 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:56.991634 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:56.991589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:58.431696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:58.431656 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:58.946139 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:58.946102 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:07:58.949025 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:58.949001 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:07:58.949277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:07:58.949252 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" containerID="cri-o://5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026" gracePeriod=30 Nov 03 00:08:00.221947 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:00.221901 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:08:00.221947 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:00.221922 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:08:00.482703 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:00.482617 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:03.677441 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:03.677404 4494 patch_prober.go:29] interesting pod/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:08:03.677847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:03.677488 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:08:04.222426 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.222361 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:08:04.388455 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388427 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388523 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388578 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388615 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388645 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388936 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388684 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388936 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388783 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388936 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388858 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.388936 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388927 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\") pod \"f8414450-c908-4418-8983-2f2f9efe1695\" (UID: \"f8414450-c908-4418-8983-2f2f9efe1695\") " Nov 03 00:08:04.389152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.388982 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.389152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389018 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.389152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389047 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:04.389152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389059 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.389438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389216 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.389438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389387 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.389438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389404 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8414450-c908-4418-8983-2f2f9efe1695-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.389438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389418 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.389438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389431 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.389854 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.389443 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.391224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.391196 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.391352 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.391316 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:08:04.391440 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.391418 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:04.391496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.391435 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk" pod "f8414450-c908-4418-8983-2f2f9efe1695" (UID: "f8414450-c908-4418-8983-2f2f9efe1695"). InnerVolumeSpecName "kube-api-access-gzkhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:04.490869 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.490780 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8414450-c908-4418-8983-2f2f9efe1695-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.490869 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.490811 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-kube-api-access-gzkhk\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.490869 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.490824 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8414450-c908-4418-8983-2f2f9efe1695-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.490869 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.490836 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8414450-c908-4418-8983-2f2f9efe1695-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:04.613160 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.613120 4494 generic.go:356] "Generic (PLEG): container finished" podID="f8414450-c908-4418-8983-2f2f9efe1695" containerID="5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026" exitCode=0 Nov 03 00:08:04.613372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.613204 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" Nov 03 00:08:04.613372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.613200 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" event={"ID":"f8414450-c908-4418-8983-2f2f9efe1695","Type":"ContainerDied","Data":"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026"} Nov 03 00:08:04.613372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.613330 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq" event={"ID":"f8414450-c908-4418-8983-2f2f9efe1695","Type":"ContainerDied","Data":"f934fd20a296784099d63c187d2834badb737cfa161b78aae2527dd5f6053a60"} Nov 03 00:08:04.613372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.613358 4494 scope.go:122] "RemoveContainer" containerID="5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026" Nov 03 00:08:04.624647 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.624629 4494 scope.go:122] "RemoveContainer" containerID="5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026" Nov 03 00:08:04.624923 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:04.624900 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026\": container with ID starting with 5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026 not found: ID does not exist" containerID="5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026" Nov 03 00:08:04.624999 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.624935 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026"} err="failed to get container status \"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026\": rpc error: code = NotFound desc = could not find container \"5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026\": container with ID starting with 5d7e134c8c77bf6683aed2bbe36e2257a2a045db4ed4d04ddd933a08ad0d2026 not found: ID does not exist" Nov 03 00:08:04.780503 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.780408 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:08:04.784233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:04.784206 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-96031a-openshift-default-5576c587f-svvhq"] Nov 03 00:08:05.550777 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:05.550739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:05.693448 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:05.693415 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8414450-c908-4418-8983-2f2f9efe1695" path="/var/lib/kubelet/pods/f8414450-c908-4418-8983-2f2f9efe1695/volumes" Nov 03 00:08:07.321046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.317877 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 03 00:08:07.412006 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.411982 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 03 00:08:07.428499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.428456 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\" (UniqueName: \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\") pod \"c7815790-5819-412d-a30d-dd0541587c3b\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " Nov 03 00:08:07.428666 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.428505 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\") pod \"091c3f43-8afb-40cb-978c-fffd8b2bd867\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " Nov 03 00:08:07.428666 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.428601 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\") pod \"c7815790-5819-412d-a30d-dd0541587c3b\" (UID: \"c7815790-5819-412d-a30d-dd0541587c3b\") " Nov 03 00:08:07.428937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.428910 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume" pod "091c3f43-8afb-40cb-978c-fffd8b2bd867" (UID: "091c3f43-8afb-40cb-978c-fffd8b2bd867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:07.429040 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.428918 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume" pod "c7815790-5819-412d-a30d-dd0541587c3b" (UID: "c7815790-5819-412d-a30d-dd0541587c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:07.430706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.430686 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg" pod "c7815790-5819-412d-a30d-dd0541587c3b" (UID: "c7815790-5819-412d-a30d-dd0541587c3b"). InnerVolumeSpecName "kube-api-access-c2srg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:07.529944 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.529904 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\" (UniqueName: \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\") pod \"091c3f43-8afb-40cb-978c-fffd8b2bd867\" (UID: \"091c3f43-8afb-40cb-978c-fffd8b2bd867\") " Nov 03 00:08:07.530349 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.530328 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2srg\" (UniqueName: \"kubernetes.io/projected/c7815790-5819-412d-a30d-dd0541587c3b-kube-api-access-c2srg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:07.530438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.530352 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091c3f43-8afb-40cb-978c-fffd8b2bd867-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:07.530438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.530365 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7815790-5819-412d-a30d-dd0541587c3b-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:07.532100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.532078 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9" pod "091c3f43-8afb-40cb-978c-fffd8b2bd867" (UID: "091c3f43-8afb-40cb-978c-fffd8b2bd867"). InnerVolumeSpecName "kube-api-access-5kpm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:07.632062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.631972 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kpm9\" (UniqueName: \"kubernetes.io/projected/091c3f43-8afb-40cb-978c-fffd8b2bd867-kube-api-access-5kpm9\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:07.642934 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.642904 4494 generic.go:356] "Generic (PLEG): container finished" podID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerID="515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88" exitCode=0 Nov 03 00:08:07.643089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.642958 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" Nov 03 00:08:07.643089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.642987 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" event={"ID":"091c3f43-8afb-40cb-978c-fffd8b2bd867","Type":"ContainerDied","Data":"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88"} Nov 03 00:08:07.643089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.643069 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7" event={"ID":"091c3f43-8afb-40cb-978c-fffd8b2bd867","Type":"ContainerDied","Data":"eb276ec1afd22b01787e2ee7dea15ba4c52e0d2ec8bd85eafff6d0f857b7b96e"} Nov 03 00:08:07.643278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.643093 4494 scope.go:122] "RemoveContainer" containerID="515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88" Nov 03 00:08:07.644498 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.644479 4494 generic.go:356] "Generic (PLEG): container finished" podID="c7815790-5819-412d-a30d-dd0541587c3b" containerID="67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592" exitCode=0 Nov 03 00:08:07.644623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.644536 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" Nov 03 00:08:07.644623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.644561 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" event={"ID":"c7815790-5819-412d-a30d-dd0541587c3b","Type":"ContainerDied","Data":"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592"} Nov 03 00:08:07.644623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.644615 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2" event={"ID":"c7815790-5819-412d-a30d-dd0541587c3b","Type":"ContainerDied","Data":"09bbe2ae73a4eaca55ea46ec76ccd63170efe65135dcdf76a042d546ec789931"} Nov 03 00:08:07.655084 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.655066 4494 scope.go:122] "RemoveContainer" containerID="515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88" Nov 03 00:08:07.655470 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:07.655450 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88\": container with ID starting with 515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88 not found: ID does not exist" containerID="515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88" Nov 03 00:08:07.655554 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.655479 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88"} err="failed to get container status \"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88\": rpc error: code = NotFound desc = could not find container \"515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88\": container with ID starting with 515b162e0dc142d35e96a0443dffc862196cb2cecec8009af8958a1b799cfe88 not found: ID does not exist" Nov 03 00:08:07.655554 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.655496 4494 scope.go:122] "RemoveContainer" containerID="67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592" Nov 03 00:08:07.664570 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.664553 4494 scope.go:122] "RemoveContainer" containerID="67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592" Nov 03 00:08:07.664871 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:07.664851 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592\": container with ID starting with 67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592 not found: ID does not exist" containerID="67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592" Nov 03 00:08:07.664923 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.664879 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592"} err="failed to get container status \"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592\": rpc error: code = NotFound desc = could not find container \"67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592\": container with ID starting with 67aa360c9db7bdb0966f88cbf500027e026a8cc9af907c2ef3e32946de014592 not found: ID does not exist" Nov 03 00:08:07.809032 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.809000 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 03 00:08:07.813079 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.812996 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-882a2a-7b796656f-46rc2"] Nov 03 00:08:07.922907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.922874 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 03 00:08:07.926877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:07.926845 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-882a2a-546648565f-d5nf7"] Nov 03 00:08:09.693184 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:09.693139 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" path="/var/lib/kubelet/pods/091c3f43-8afb-40cb-978c-fffd8b2bd867/volumes" Nov 03 00:08:09.693592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:09.693581 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7815790-5819-412d-a30d-dd0541587c3b" path="/var/lib/kubelet/pods/c7815790-5819-412d-a30d-dd0541587c3b/volumes" Nov 03 00:08:10.538307 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:10.538277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:15.383083 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:15.383049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:17.559596 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:17.559511 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:08:20.241360 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:20.241327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:22.968956 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:22.968919 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:25.137299 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:25.137272 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:25.973694 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:25.973651 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:08:28.705732 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:28.705698 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:29.360483 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.360446 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 03 00:08:29.360689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.360665 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" podUID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" containerName="maas-api" containerID="cri-o://908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c" gracePeriod=30 Nov 03 00:08:29.464591 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.464549 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:29.662561 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.662535 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 03 00:08:29.795598 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.795563 4494 generic.go:356] "Generic (PLEG): container finished" podID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" containerID="908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c" exitCode=0 Nov 03 00:08:29.796106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.795617 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" Nov 03 00:08:29.796106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.795645 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" event={"ID":"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6","Type":"ContainerDied","Data":"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c"} Nov 03 00:08:29.796106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.795702 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx" event={"ID":"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6","Type":"ContainerDied","Data":"84797508bd602c0b92ecbf3fee6c3b678ce10f83d25a44424f226b8e0183562b"} Nov 03 00:08:29.796106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.795726 4494 scope.go:122] "RemoveContainer" containerID="908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c" Nov 03 00:08:29.806742 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.806645 4494 scope.go:122] "RemoveContainer" containerID="908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c" Nov 03 00:08:29.806955 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:29.806931 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c\": container with ID starting with 908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c not found: ID does not exist" containerID="908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c" Nov 03 00:08:29.807071 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.806963 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c"} err="failed to get container status \"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c\": rpc error: code = NotFound desc = could not find container \"908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c\": container with ID starting with 908c0fb3e713a101b3b63e9c580c61c53d2fc0a7a7402a842835cfc402f5de1c not found: ID does not exist" Nov 03 00:08:29.807071 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.806995 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\" (UniqueName: \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\") pod \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " Nov 03 00:08:29.807206 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.807112 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") pod \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " Nov 03 00:08:29.807253 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.807165 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\") pod \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " Nov 03 00:08:29.807377 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.807353 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\") pod \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\" (UID: \"16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6\") " Nov 03 00:08:29.807692 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.807645 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca" pod "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" (UID: "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:29.808276 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.808233 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:29.812216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.809753 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs" pod "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" (UID: "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:08:29.812216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.809903 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls" pod "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" (UID: "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:08:29.812216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.809975 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs" pod "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" (UID: "16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6"). InnerVolumeSpecName "kube-api-access-c9hjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:29.913907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.910348 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:29.913907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.910382 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:29.913907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:29.910397 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9hjs\" (UniqueName: \"kubernetes.io/projected/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6-kube-api-access-c9hjs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:30.219562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:30.219473 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 03 00:08:30.223729 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:30.223694 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-882a2a-6fb777fb9b-c6sbx"] Nov 03 00:08:31.243237 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:31.243199 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:31.694131 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:31.694099 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" path="/var/lib/kubelet/pods/16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6/volumes" Nov 03 00:08:32.649207 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:32.648291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:32.793861 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:32.793809 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:34.151043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:34.151009 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:34.656570 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:34.656527 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:34.660609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:34.660582 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 03 00:08:34.660810 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:34.660790 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" containerID="cri-o://cd974d1c5712cc8e06704ed5533a86838504584622a7770ac837db75f70a583e" gracePeriod=30 Nov 03 00:08:35.479692 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:35.479651 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:36.272495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:36.272463 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:37.069895 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:37.069865 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:39.877398 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:39.877362 4494 generic.go:356] "Generic (PLEG): container finished" podID="3640416e-01aa-4410-b91e-8937ad196b86" containerID="cd974d1c5712cc8e06704ed5533a86838504584622a7770ac837db75f70a583e" exitCode=0 Nov 03 00:08:39.877845 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:39.877425 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerDied","Data":"cd974d1c5712cc8e06704ed5533a86838504584622a7770ac837db75f70a583e"} Nov 03 00:08:39.877845 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:39.877489 4494 scope.go:122] "RemoveContainer" containerID="2587eb675d601428f2bf03c6e13894d0f4ef2851be554a690cbb88fdcef15d9e" Nov 03 00:08:39.925453 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:39.925396 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 03 00:08:40.040862 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.040827 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041072 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.040901 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041072 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.040983 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041088 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041191 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041253 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041287 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041312 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:40.041362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041320 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041349 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:40.041583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041359 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:40.041583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041384 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\") pod \"3640416e-01aa-4410-b91e-8937ad196b86\" (UID: \"3640416e-01aa-4410-b91e-8937ad196b86\") " Nov 03 00:08:40.041583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041496 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:40.041583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041506 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:40.041842 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041823 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.041953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041844 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.041953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041852 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.041953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041861 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3640416e-01aa-4410-b91e-8937ad196b86-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.041953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.041870 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.043583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.043561 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:40.043696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.043595 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "kube-api-access-pffr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:40.043958 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.043936 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:08:40.044031 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.043969 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token" pod "3640416e-01aa-4410-b91e-8937ad196b86" (UID: "3640416e-01aa-4410-b91e-8937ad196b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:40.142897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.142866 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3640416e-01aa-4410-b91e-8937ad196b86-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.142897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.142893 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffr5\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-kube-api-access-pffr5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.142897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.142904 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3640416e-01aa-4410-b91e-8937ad196b86-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.143133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.142912 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3640416e-01aa-4410-b91e-8937ad196b86-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.884310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.884281 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" Nov 03 00:08:40.884816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.884288 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv" event={"ID":"3640416e-01aa-4410-b91e-8937ad196b86","Type":"ContainerDied","Data":"206d49d7fee50089442640378080ed7aff73c345a794927b3a5534b0de6e143f"} Nov 03 00:08:40.884816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:40.884405 4494 scope.go:122] "RemoveContainer" containerID="cd974d1c5712cc8e06704ed5533a86838504584622a7770ac837db75f70a583e" Nov 03 00:08:41.102612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:41.102561 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 03 00:08:41.153330 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:41.153245 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-882a2a-openshift-default-fbc7c88bc-qgcxv"] Nov 03 00:08:41.332982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:41.332940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:41.693946 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:41.693904 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3640416e-01aa-4410-b91e-8937ad196b86" path="/var/lib/kubelet/pods/3640416e-01aa-4410-b91e-8937ad196b86/volumes" Nov 03 00:08:42.221732 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:42.221671 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:43.177431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:43.177401 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:43.323820 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:43.323784 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:46.538741 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:46.538708 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:47.559952 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:08:47.559907 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:08:47.793909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:47.793874 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:51.425025 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:51.424971 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:55.150350 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:55.150310 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:56.351620 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:56.351576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:08:57.559460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:08:57.559424 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:09:01.215883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:01.215848 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:06.139807 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:06.139770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:11.016433 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:11.016385 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:15.880941 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:15.880908 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:17.559483 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:09:17.559439 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:09:20.162064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:20.162032 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:20.853304 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:20.853263 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 03 00:09:20.866198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:20.861230 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" containerID="cri-o://88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525" gracePeriod=30 Nov 03 00:09:21.939580 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:21.939547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:23.380624 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:23.380589 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:26.141455 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.141425 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 03 00:09:26.189424 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.189356 4494 generic.go:356] "Generic (PLEG): container finished" podID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerID="88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525" exitCode=0 Nov 03 00:09:26.189424 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.189398 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" Nov 03 00:09:26.189424 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.189400 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" event={"ID":"159a248d-7ffa-4fa5-b2f6-11690ecac868","Type":"ContainerDied","Data":"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525"} Nov 03 00:09:26.189633 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.189445 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p" event={"ID":"159a248d-7ffa-4fa5-b2f6-11690ecac868","Type":"ContainerDied","Data":"317de9bfa61169ff4f1adfe89f6dabbe52b012958826cad6c1fd6b35444ff201"} Nov 03 00:09:26.189633 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.189467 4494 scope.go:122] "RemoveContainer" containerID="88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525" Nov 03 00:09:26.198837 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.198819 4494 scope.go:122] "RemoveContainer" containerID="88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525" Nov 03 00:09:26.199137 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:09:26.199119 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525\": container with ID starting with 88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525 not found: ID does not exist" containerID="88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525" Nov 03 00:09:26.199205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.199145 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525"} err="failed to get container status \"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525\": rpc error: code = NotFound desc = could not find container \"88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525\": container with ID starting with 88849a156340076ade999616902469809c539316c2a53f7c7eef3a36b9e68525 not found: ID does not exist" Nov 03 00:09:26.296094 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296061 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296094 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296098 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296121 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296219 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296258 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296319 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296356 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296423 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296438 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:26.296706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296564 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\") pod \"159a248d-7ffa-4fa5-b2f6-11690ecac868\" (UID: \"159a248d-7ffa-4fa5-b2f6-11690ecac868\") " Nov 03 00:09:26.296706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296655 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:26.296938 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.296918 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:26.297064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.297044 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.297064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.297048 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:26.297478 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.297071 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.297478 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.297086 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159a248d-7ffa-4fa5-b2f6-11690ecac868-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.297478 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.297114 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:26.299112 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.299080 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:26.299605 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.299577 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:09:26.299702 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.299603 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:09:26.299886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.299859 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv" pod "159a248d-7ffa-4fa5-b2f6-11690ecac868" (UID: "159a248d-7ffa-4fa5-b2f6-11690ecac868"). InnerVolumeSpecName "kube-api-access-ttrcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399222 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399262 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrcv\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-kube-api-access-ttrcv\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399279 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399292 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399306 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.399347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.399318 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159a248d-7ffa-4fa5-b2f6-11690ecac868-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:26.612038 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.612003 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 03 00:09:26.617763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.617738 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ql99p"] Nov 03 00:09:26.840225 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:26.840114 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:27.693821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:27.693789 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" path="/var/lib/kubelet/pods/159a248d-7ffa-4fa5-b2f6-11690ecac868/volumes" Nov 03 00:09:28.790096 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:28.790059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:28.934302 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:28.934241 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:30.334071 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:30.334033 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:30.819643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:30.819604 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:31.302256 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:31.302224 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:31.963277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:31.963243 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:34.356597 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:34.356566 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:09:35.129882 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:35.129848 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:35.276123 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:35.276083 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:36.303877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:36.303839 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:36.452490 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:36.452449 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:37.054639 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:37.054611 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:38.428060 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:38.428011 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:41.953906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:41.953877 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:43.935363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:43.935321 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 03 00:09:44.081538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:44.081505 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 03 00:09:44.338516 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:44.338436 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:44.489334 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:44.489303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:44.964584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:44.964554 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:47.222631 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:47.222597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:47.560457 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:09:47.560349 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:09:47.890053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:47.889965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:48.935662 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:48.935621 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" podUID="2259f9bc-a646-45b1-9ba4-71ae5038d540" containerName="payload-processing" containerID="cri-o://7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68" gracePeriod=25 Nov 03 00:09:49.081987 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.081925 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" podUID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" containerName="payload-pre-processing" containerID="cri-o://2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578" gracePeriod=25 Nov 03 00:09:49.211466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.211443 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 03 00:09:49.334249 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.332140 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 03 00:09:49.336119 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.336060 4494 generic.go:356] "Generic (PLEG): container finished" podID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" containerID="2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578" exitCode=0 Nov 03 00:09:49.336328 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.336307 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" event={"ID":"726e5179-b6a5-48d5-a8c3-a570d90e97f2","Type":"ContainerDied","Data":"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578"} Nov 03 00:09:49.336447 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.336430 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" event={"ID":"726e5179-b6a5-48d5-a8c3-a570d90e97f2","Type":"ContainerDied","Data":"58cf7040f7aa397e20c1e8ccbedd9cf9028db8e23b2eb62d920a564a13710ecd"} Nov 03 00:09:49.336523 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.336474 4494 scope.go:122] "RemoveContainer" containerID="2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578" Nov 03 00:09:49.336719 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.336705 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl" Nov 03 00:09:49.340455 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.340428 4494 generic.go:356] "Generic (PLEG): container finished" podID="2259f9bc-a646-45b1-9ba4-71ae5038d540" containerID="7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68" exitCode=0 Nov 03 00:09:49.340624 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.340500 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" Nov 03 00:09:49.340755 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.340722 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" event={"ID":"2259f9bc-a646-45b1-9ba4-71ae5038d540","Type":"ContainerDied","Data":"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68"} Nov 03 00:09:49.340886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.340786 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl" event={"ID":"2259f9bc-a646-45b1-9ba4-71ae5038d540","Type":"ContainerDied","Data":"93c26a243945c486270566b0d27d49e1163b0948a7f1a0dfa45bdafba93df99b"} Nov 03 00:09:49.353528 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.353495 4494 scope.go:122] "RemoveContainer" containerID="2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578" Nov 03 00:09:49.353925 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:09:49.353884 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578\": container with ID starting with 2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578 not found: ID does not exist" containerID="2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578" Nov 03 00:09:49.354103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.353932 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578"} err="failed to get container status \"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578\": rpc error: code = NotFound desc = could not find container \"2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578\": container with ID starting with 2cc1dc271ed76e7744f4256b32490e3f8102e0a9934014bac834a9bebea98578 not found: ID does not exist" Nov 03 00:09:49.354103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.353954 4494 scope.go:122] "RemoveContainer" containerID="7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68" Nov 03 00:09:49.364315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.364296 4494 scope.go:122] "RemoveContainer" containerID="7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68" Nov 03 00:09:49.364638 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:09:49.364606 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68\": container with ID starting with 7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68 not found: ID does not exist" containerID="7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68" Nov 03 00:09:49.364713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.364652 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68"} err="failed to get container status \"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68\": rpc error: code = NotFound desc = could not find container \"7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68\": container with ID starting with 7d62345595c547d47df3b4d334560e66baa51aa14641df21da47493a5c87ad68 not found: ID does not exist" Nov 03 00:09:49.377546 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.377525 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\") pod \"2259f9bc-a646-45b1-9ba4-71ae5038d540\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " Nov 03 00:09:49.377646 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.377633 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\" (UniqueName: \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\") pod \"2259f9bc-a646-45b1-9ba4-71ae5038d540\" (UID: \"2259f9bc-a646-45b1-9ba4-71ae5038d540\") " Nov 03 00:09:49.377868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.377849 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume" pod "2259f9bc-a646-45b1-9ba4-71ae5038d540" (UID: "2259f9bc-a646-45b1-9ba4-71ae5038d540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:49.379908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.379885 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh" pod "2259f9bc-a646-45b1-9ba4-71ae5038d540" (UID: "2259f9bc-a646-45b1-9ba4-71ae5038d540"). InnerVolumeSpecName "kube-api-access-kxjsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:49.479073 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.478971 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\" (UniqueName: \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\") pod \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " Nov 03 00:09:49.479073 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.479067 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\") pod \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\" (UID: \"726e5179-b6a5-48d5-a8c3-a570d90e97f2\") " Nov 03 00:09:49.479421 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.479406 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2259f9bc-a646-45b1-9ba4-71ae5038d540-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:49.479499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.479422 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjsh\" (UniqueName: \"kubernetes.io/projected/2259f9bc-a646-45b1-9ba4-71ae5038d540-kube-api-access-kxjsh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:49.479821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.479793 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume" pod "726e5179-b6a5-48d5-a8c3-a570d90e97f2" (UID: "726e5179-b6a5-48d5-a8c3-a570d90e97f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:09:49.481396 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.481368 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc" pod "726e5179-b6a5-48d5-a8c3-a570d90e97f2" (UID: "726e5179-b6a5-48d5-a8c3-a570d90e97f2"). InnerVolumeSpecName "kube-api-access-kk5rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:09:49.580399 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.580363 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5rc\" (UniqueName: \"kubernetes.io/projected/726e5179-b6a5-48d5-a8c3-a570d90e97f2-kube-api-access-kk5rc\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:49.580399 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.580393 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726e5179-b6a5-48d5-a8c3-a570d90e97f2-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:09:49.760638 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.760552 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 03 00:09:49.764775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.764751 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-637fdb-b5996b99-6kfgl"] Nov 03 00:09:49.875287 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.875251 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 03 00:09:49.881611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:49.881586 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-637fdb-5d5c55956c-4h2fl"] Nov 03 00:09:50.860155 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:50.860120 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:51.693324 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:51.693291 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2259f9bc-a646-45b1-9ba4-71ae5038d540" path="/var/lib/kubelet/pods/2259f9bc-a646-45b1-9ba4-71ae5038d540/volumes" Nov 03 00:09:51.693710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:51.693698 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" path="/var/lib/kubelet/pods/726e5179-b6a5-48d5-a8c3-a570d90e97f2/volumes" Nov 03 00:09:52.362832 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:52.362801 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:56.924146 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:56.924106 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:09:57.350462 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:09:57.350378 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:02.351073 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:02.351031 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:04.031000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:04.030967 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:10:07.262473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:07.262438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:12.240303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:12.240265 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:14.199937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:14.199889 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:14.344479 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:14.344428 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:14.495556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:14.495465 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:17.159843 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:17.159814 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:17.483011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:17.482918 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:17.559137 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:10:17.559102 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:10:17.630466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:17.630433 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:20.503655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:20.503626 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:20.652373 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:20.652339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:22.048254 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:22.048217 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:23.656357 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:23.656314 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:26.953451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:26.953419 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:31.229890 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:31.229846 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:33.033514 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:33.033483 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:37.951535 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:37.951499 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.434587 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:38.434550 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:38.581993 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:38.581964 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:40.040649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:40.040619 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:40.402396 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:40.402312 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:10:40.547481 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:40.547451 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:43.439489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:43.439452 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:47.559585 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:10:47.559547 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:10:48.445881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:48.445818 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:53.350811 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:53.350779 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:10:58.263343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:10:58.263309 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:01.613892 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:01.613860 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:11:03.152355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:03.152320 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:08.047845 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:08.047811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:12.947493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:12.947453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:17.560004 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:11:17.559909 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:11:17.851839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:17.851741 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:18.363409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:18.363381 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:18.514083 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:18.514049 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:23.762755 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:23.762711 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:24.002302 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:24.002259 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:24.765095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:24.765062 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:24.907609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:24.907562 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:27.635508 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:27.635478 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:28.850955 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:28.850920 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:30.274931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:30.274891 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:32.630531 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:32.630493 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:34.006098 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:34.006061 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:11:37.465942 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:37.465894 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:42.346885 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:42.346853 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:47.256013 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:47.255975 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:47.560310 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:11:47.560218 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:11:52.151647 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:52.151609 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:11:57.046650 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:11:57.046614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:01.947068 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:01.947030 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:06.580326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:06.580294 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:12:06.833010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:06.832932 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:09.170106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:09.170062 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:11.555554 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:11.555508 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:11.742410 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:11.742375 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:11.996928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:11.996888 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:12.140719 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:12.140682 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:16.642564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:16.642530 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:17.558557 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:12:17.558520 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:12:18.640633 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:18.640597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:21.534605 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:21.534564 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:24.416763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:24.416727 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:26.424073 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:26.424037 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:31.359964 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:31.359928 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:33.436493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:33.436458 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:12:36.235330 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:36.235295 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:39.382003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:39.381965 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:40.003031 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:40.002997 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:40.451206 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:40.451161 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:41.161434 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:41.161401 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:46.044008 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:46.043976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:46.342658 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:46.342569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:46.938239 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:46.938203 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 03 00:12:47.299954 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:47.299863 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:47.468902 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:47.468855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:47.558382 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:12:47.558274 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:12:51.247330 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:51.247296 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:51.528258 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:51.528160 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:12:51.938935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:51.938896 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" containerID="cri-o://5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7" gracePeriod=25 Nov 03 00:12:52.500630 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:52.500591 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:54.467686 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:54.467653 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:12:56.147372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:56.147340 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:56.827565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:56.827532 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:12:56.972566 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:12:56.972532 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:01.103497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:01.103444 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:04.476442 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:04.476392 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:05.973162 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:05.973125 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:10.858486 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:10.858453 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:12.190723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.190702 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 03 00:13:12.268022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.267946 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\") pod \"384a4b01-50be-41cb-9cff-2bf4ada7592d\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " Nov 03 00:13:12.268209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.268109 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\" (UniqueName: \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\") pod \"384a4b01-50be-41cb-9cff-2bf4ada7592d\" (UID: \"384a4b01-50be-41cb-9cff-2bf4ada7592d\") " Nov 03 00:13:12.268370 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.268348 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume" pod "384a4b01-50be-41cb-9cff-2bf4ada7592d" (UID: "384a4b01-50be-41cb-9cff-2bf4ada7592d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:12.270363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.270339 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5" pod "384a4b01-50be-41cb-9cff-2bf4ada7592d" (UID: "384a4b01-50be-41cb-9cff-2bf4ada7592d"). InnerVolumeSpecName "kube-api-access-x7vv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:12.369043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.369013 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7vv5\" (UniqueName: \"kubernetes.io/projected/384a4b01-50be-41cb-9cff-2bf4ada7592d-kube-api-access-x7vv5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:12.369043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.369038 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a4b01-50be-41cb-9cff-2bf4ada7592d-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:12.509940 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.509901 4494 generic.go:356] "Generic (PLEG): container finished" podID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerID="5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7" exitCode=0 Nov 03 00:13:12.510151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.509952 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" Nov 03 00:13:12.510151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.509989 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" event={"ID":"384a4b01-50be-41cb-9cff-2bf4ada7592d","Type":"ContainerDied","Data":"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7"} Nov 03 00:13:12.510151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.510048 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2" event={"ID":"384a4b01-50be-41cb-9cff-2bf4ada7592d","Type":"ContainerDied","Data":"8d28d99ede24448cbcbf66b7f13abfa735dfba55c856fadfc01c9dae4c1ee196"} Nov 03 00:13:12.510151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.510077 4494 scope.go:122] "RemoveContainer" containerID="5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7" Nov 03 00:13:12.520547 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.520528 4494 scope.go:122] "RemoveContainer" containerID="5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7" Nov 03 00:13:12.520845 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:12.520822 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7\": container with ID starting with 5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7 not found: ID does not exist" containerID="5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7" Nov 03 00:13:12.520916 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.520857 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7"} err="failed to get container status \"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7\": rpc error: code = NotFound desc = could not find container \"5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7\": container with ID starting with 5fb3b612687bf40165f7bd2f332d5c8a2dca57d33aea6680bc3bf5451c3880d7 not found: ID does not exist" Nov 03 00:13:12.675945 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.675911 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 03 00:13:12.679505 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:12.679478 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-980b82-589964dd8c-s6vw2"] Nov 03 00:13:13.693102 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:13.693057 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" path="/var/lib/kubelet/pods/384a4b01-50be-41cb-9cff-2bf4ada7592d/volumes" Nov 03 00:13:15.747047 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:15.747011 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:17.559932 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:17.559894 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:13:19.667974 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:19.667940 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:20.032411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:20.032327 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:21.754770 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:21.754737 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:24.002576 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:24.002545 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:24.150411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:24.150374 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:24.698850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:24.698811 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:25.556330 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:25.556297 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:26.225919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:26.225866 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:26.228931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:26.228910 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 03 00:13:26.229121 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:26.229100 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" containerID="cri-o://9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933" gracePeriod=30 Nov 03 00:13:26.931139 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:26.931102 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:27.076447 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:27.076414 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:27.527246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:27.527212 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:13:31.496226 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.496147 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 03 00:13:31.591640 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591609 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.591828 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591657 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.591828 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591694 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.591828 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591805 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591842 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591906 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591951 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.591997 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592068 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:31.592251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592141 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:31.592345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592279 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:31.592403 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592384 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\") pod \"4c8426ee-2058-4c4a-b96d-c503316a01bf\" (UID: \"4c8426ee-2058-4c4a-b96d-c503316a01bf\") " Nov 03 00:13:31.592456 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592417 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:31.592710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592689 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:31.592943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592925 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.593021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592948 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.593021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592963 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8426ee-2058-4c4a-b96d-c503316a01bf-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.593021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592977 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.593021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.592990 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.594025 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.594005 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:31.594651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.594633 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:13:31.594745 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.594723 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:31.594786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.594753 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r" pod "4c8426ee-2058-4c4a-b96d-c503316a01bf" (UID: "4c8426ee-2058-4c4a-b96d-c503316a01bf"). InnerVolumeSpecName "kube-api-access-x9v8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:31.617219 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.617196 4494 generic.go:356] "Generic (PLEG): container finished" podID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerID="9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933" exitCode=0 Nov 03 00:13:31.617328 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.617244 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" Nov 03 00:13:31.617328 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.617252 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerDied","Data":"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933"} Nov 03 00:13:31.617328 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.617318 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj" event={"ID":"4c8426ee-2058-4c4a-b96d-c503316a01bf","Type":"ContainerDied","Data":"a21406ea52f19c6a30515e88dcb4488e07059d93c183fc1d5979a338e15b547b"} Nov 03 00:13:31.617442 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.617338 4494 scope.go:122] "RemoveContainer" containerID="9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933" Nov 03 00:13:31.631064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.631036 4494 scope.go:122] "RemoveContainer" containerID="aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc" Nov 03 00:13:31.640784 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.640761 4494 scope.go:122] "RemoveContainer" containerID="9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933" Nov 03 00:13:31.641048 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:31.641026 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933\": container with ID starting with 9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933 not found: ID does not exist" containerID="9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933" Nov 03 00:13:31.641136 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.641057 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933"} err="failed to get container status \"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933\": rpc error: code = NotFound desc = could not find container \"9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933\": container with ID starting with 9040d1011df960fcd5bdfd25bef368bd67f058389a5b368e42984d805297d933 not found: ID does not exist" Nov 03 00:13:31.641136 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.641075 4494 scope.go:122] "RemoveContainer" containerID="aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc" Nov 03 00:13:31.641391 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:31.641370 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc\": container with ID starting with aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc not found: ID does not exist" containerID="aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc" Nov 03 00:13:31.641451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.641398 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc"} err="failed to get container status \"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc\": rpc error: code = NotFound desc = could not find container \"aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc\": container with ID starting with aaac31c47da601c4251ef8cb252dd4242110114f31e5c7fb60139e2e77bf98dc not found: ID does not exist" Nov 03 00:13:31.694578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.694551 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9v8r\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-kube-api-access-x9v8r\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.694578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.694576 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.694783 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.694588 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.694783 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.694597 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8426ee-2058-4c4a-b96d-c503316a01bf-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:31.780968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.780874 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 03 00:13:31.782961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:31.782939 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-980b82-openshift-default-6fbf64f657-g7fhj"] Nov 03 00:13:32.079277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:32.079163 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:32.279947 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:32.279905 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 03 00:13:32.640771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:32.640739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:32.844556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:32.844522 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:32.988881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:32.988839 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:33.692855 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:33.692822 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" path="/var/lib/kubelet/pods/4c8426ee-2058-4c4a-b96d-c503316a01bf/volumes" Nov 03 00:13:37.280652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:37.280609 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" containerID="cri-o://da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba" gracePeriod=25 Nov 03 00:13:37.345201 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:37.345129 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:38.946240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:38.946200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:42.257086 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:42.257046 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:47.127569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:47.127531 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:47.150809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:47.150769 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:47.558616 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:47.558581 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:13:52.052588 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:52.052548 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:56.250686 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.250651 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 03 00:13:56.251225 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.250866 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" podUID="967731db-2140-4a0c-a9d3-2272b862aa67" containerName="maas-api" containerID="cri-o://341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381" gracePeriod=30 Nov 03 00:13:56.339209 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.339139 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:56.540914 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.540888 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 03 00:13:56.678143 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.678112 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\") pod \"967731db-2140-4a0c-a9d3-2272b862aa67\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " Nov 03 00:13:56.678414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.678246 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\") pod \"967731db-2140-4a0c-a9d3-2272b862aa67\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " Nov 03 00:13:56.678414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.678277 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") pod \"967731db-2140-4a0c-a9d3-2272b862aa67\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " Nov 03 00:13:56.678414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.678297 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\" (UniqueName: \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\") pod \"967731db-2140-4a0c-a9d3-2272b862aa67\" (UID: \"967731db-2140-4a0c-a9d3-2272b862aa67\") " Nov 03 00:13:56.678663 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.678633 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca" pod "967731db-2140-4a0c-a9d3-2272b862aa67" (UID: "967731db-2140-4a0c-a9d3-2272b862aa67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:56.680557 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.680526 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs" pod "967731db-2140-4a0c-a9d3-2272b862aa67" (UID: "967731db-2140-4a0c-a9d3-2272b862aa67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:56.680701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.680583 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql" pod "967731db-2140-4a0c-a9d3-2272b862aa67" (UID: "967731db-2140-4a0c-a9d3-2272b862aa67"). InnerVolumeSpecName "kube-api-access-2s4ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:56.680775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.680757 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls" pod "967731db-2140-4a0c-a9d3-2272b862aa67" (UID: "967731db-2140-4a0c-a9d3-2272b862aa67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:56.755252 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.755217 4494 generic.go:356] "Generic (PLEG): container finished" podID="967731db-2140-4a0c-a9d3-2272b862aa67" containerID="341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381" exitCode=0 Nov 03 00:13:56.755443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.755274 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" Nov 03 00:13:56.755443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.755304 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" event={"ID":"967731db-2140-4a0c-a9d3-2272b862aa67","Type":"ContainerDied","Data":"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381"} Nov 03 00:13:56.755443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.755368 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6" event={"ID":"967731db-2140-4a0c-a9d3-2272b862aa67","Type":"ContainerDied","Data":"6751e05ace7baf1b0c0dbdeebbb9cd9210c6925e9928b8ec374581f4149c40ba"} Nov 03 00:13:56.755443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.755397 4494 scope.go:122] "RemoveContainer" containerID="341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381" Nov 03 00:13:56.767615 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.767599 4494 scope.go:122] "RemoveContainer" containerID="341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381" Nov 03 00:13:56.767952 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:56.767920 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381\": container with ID starting with 341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381 not found: ID does not exist" containerID="341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381" Nov 03 00:13:56.768047 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.767968 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381"} err="failed to get container status \"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381\": rpc error: code = NotFound desc = could not find container \"341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381\": container with ID starting with 341380f283e4d95560f61e9e5d4ae2d2ddb4c239fab63d11a79b2ff296602381 not found: ID does not exist" Nov 03 00:13:56.780141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.780087 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:56.780141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.780127 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967731db-2140-4a0c-a9d3-2272b862aa67-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:56.780689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.780159 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967731db-2140-4a0c-a9d3-2272b862aa67-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:56.780689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.780219 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4ql\" (UniqueName: \"kubernetes.io/projected/967731db-2140-4a0c-a9d3-2272b862aa67-kube-api-access-2s4ql\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:56.920662 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.920625 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 03 00:13:56.926317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:56.926290 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b8d3cb-64dd96d8bf-4c2m6"] Nov 03 00:13:57.127432 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.127348 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:13:57.543249 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.543221 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 03 00:13:57.690678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.690647 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\") pod \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " Nov 03 00:13:57.690916 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.690858 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\" (UniqueName: \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\") pod \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\" (UID: \"fc0e0549-0c5e-4a72-9814-46c74ac30bc2\") " Nov 03 00:13:57.691228 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.691162 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume" pod "fc0e0549-0c5e-4a72-9814-46c74ac30bc2" (UID: "fc0e0549-0c5e-4a72-9814-46c74ac30bc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:57.691382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.691340 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:57.693617 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.693584 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4" pod "fc0e0549-0c5e-4a72-9814-46c74ac30bc2" (UID: "fc0e0549-0c5e-4a72-9814-46c74ac30bc2"). InnerVolumeSpecName "kube-api-access-zglp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:57.693812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.693785 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967731db-2140-4a0c-a9d3-2272b862aa67" path="/var/lib/kubelet/pods/967731db-2140-4a0c-a9d3-2272b862aa67/volumes" Nov 03 00:13:57.762069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.762030 4494 generic.go:356] "Generic (PLEG): container finished" podID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerID="da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba" exitCode=0 Nov 03 00:13:57.762311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.762090 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" Nov 03 00:13:57.762311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.762110 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" event={"ID":"fc0e0549-0c5e-4a72-9814-46c74ac30bc2","Type":"ContainerDied","Data":"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba"} Nov 03 00:13:57.762311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.762184 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj" event={"ID":"fc0e0549-0c5e-4a72-9814-46c74ac30bc2","Type":"ContainerDied","Data":"21cf24c71fea51f656d9141e4469aac36e9651d2d3f5c3174c6496349830287b"} Nov 03 00:13:57.762311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.762219 4494 scope.go:122] "RemoveContainer" containerID="da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba" Nov 03 00:13:57.772999 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.772956 4494 scope.go:122] "RemoveContainer" containerID="da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba" Nov 03 00:13:57.773376 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:13:57.773341 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba\": container with ID starting with da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba not found: ID does not exist" containerID="da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba" Nov 03 00:13:57.773494 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.773386 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba"} err="failed to get container status \"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba\": rpc error: code = NotFound desc = could not find container \"da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba\": container with ID starting with da684d69bb173b999ab94248d9c16deffec8862168d6c56054d33b2d980720ba not found: ID does not exist" Nov 03 00:13:57.792972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.792942 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zglp4\" (UniqueName: \"kubernetes.io/projected/fc0e0549-0c5e-4a72-9814-46c74ac30bc2-kube-api-access-zglp4\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:13:57.928726 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.928690 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 03 00:13:57.934412 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:57.934387 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b8d3cb-984d4b5d7-pbmvj"] Nov 03 00:13:58.061798 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:58.061692 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:13:59.693001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:13:59.692969 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" path="/var/lib/kubelet/pods/fc0e0549-0c5e-4a72-9814-46c74ac30bc2/volumes" Nov 03 00:14:01.954246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:01.954197 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:02.099240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:02.099200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:03.136825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:03.136777 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:03.604113 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:03.604056 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:04.127836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:04.127795 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:04.128025 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:04.127868 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 03 00:14:04.128103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:04.128063 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" containerID="cri-o://5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4" gracePeriod=30 Nov 03 00:14:04.604327 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:04.604293 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:08.005828 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.005792 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 03 00:14:08.006336 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.006011 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" containerID="cri-o://eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32" gracePeriod=30 Nov 03 00:14:08.249602 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.249569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:08.326823 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.326733 4494 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:08.326823 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.326787 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:08.886077 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.886032 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5rkg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:08.886285 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:08.886102 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:09.389510 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.389484 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 03 00:14:09.523674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523633 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.523887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523686 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.523887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523769 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.523887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523824 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.523887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523852 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.524097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523901 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.524097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523950 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.524097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.523983 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.524097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524092 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\") pod \"48face08-0636-4c00-b7e8-9e8de6ce7983\" (UID: \"48face08-0636-4c00-b7e8-9e8de6ce7983\") " Nov 03 00:14:09.524386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524127 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.524386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524214 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:09.524386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524321 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.524599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524574 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48face08-0636-4c00-b7e8-9e8de6ce7983-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.524599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524601 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.524826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524616 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.524826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524631 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.524826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.524660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.526237 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.526213 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:09.526406 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.526387 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:09.526406 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.526395 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:09.526504 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.526477 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh" pod "48face08-0636-4c00-b7e8-9e8de6ce7983" (UID: "48face08-0636-4c00-b7e8-9e8de6ce7983"). InnerVolumeSpecName "kube-api-access-qxvjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:09.625689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625652 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.625689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625688 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvjh\" (UniqueName: \"kubernetes.io/projected/48face08-0636-4c00-b7e8-9e8de6ce7983-kube-api-access-qxvjh\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.625908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625699 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.625908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625707 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.625908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625715 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48face08-0636-4c00-b7e8-9e8de6ce7983-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.625908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.625723 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48face08-0636-4c00-b7e8-9e8de6ce7983-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:09.658777 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.658736 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:09.832896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.832805 4494 generic.go:356] "Generic (PLEG): container finished" podID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerID="5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4" exitCode=0 Nov 03 00:14:09.832896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.832860 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" Nov 03 00:14:09.832896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.832882 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerDied","Data":"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4"} Nov 03 00:14:09.833233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.832933 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4" event={"ID":"48face08-0636-4c00-b7e8-9e8de6ce7983","Type":"ContainerDied","Data":"5eacfcaedb291fd8cb6c6f38f4e362cf6ae5138504acc51344b12ee2202b4440"} Nov 03 00:14:09.833233 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.832943 4494 scope.go:122] "RemoveContainer" containerID="5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4" Nov 03 00:14:09.842231 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.842217 4494 scope.go:122] "RemoveContainer" containerID="181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945" Nov 03 00:14:09.850778 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.850760 4494 scope.go:122] "RemoveContainer" containerID="5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4" Nov 03 00:14:09.851069 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:14:09.851048 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4\": container with ID starting with 5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4 not found: ID does not exist" containerID="5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4" Nov 03 00:14:09.851160 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.851081 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4"} err="failed to get container status \"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4\": rpc error: code = NotFound desc = could not find container \"5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4\": container with ID starting with 5c1d19d976e1c802ef18bf05b93f2d16fcede1b9a58d82a8c97132852ec32de4 not found: ID does not exist" Nov 03 00:14:09.851160 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.851137 4494 scope.go:122] "RemoveContainer" containerID="181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945" Nov 03 00:14:09.851466 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:14:09.851446 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945\": container with ID starting with 181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945 not found: ID does not exist" containerID="181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945" Nov 03 00:14:09.851544 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:09.851474 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945"} err="failed to get container status \"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945\": rpc error: code = NotFound desc = could not find container \"181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945\": container with ID starting with 181217f5b074dd98af01391c4f31ff9f8f113dc28a2964bccd0df9c5874d3945 not found: ID does not exist" Nov 03 00:14:10.002603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:10.002567 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 03 00:14:10.006366 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:10.006340 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8d3cb-openshift-default-745585785-j4xc4"] Nov 03 00:14:10.444543 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:10.444504 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:14:11.692563 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:11.692532 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" path="/var/lib/kubelet/pods/48face08-0636-4c00-b7e8-9e8de6ce7983/volumes" Nov 03 00:14:11.717063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:11.717035 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:13.129813 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.129789 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:13.282068 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.282010 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 03 00:14:13.360432 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360384 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360596 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360480 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360596 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360505 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360596 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360567 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360599 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360638 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360667 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360700 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360706 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360736 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:13.360815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.360781 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\") pod \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\" (UID: \"55e32ea6-02e2-4808-ac8e-f79f2948cfeb\") " Nov 03 00:14:13.361232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361067 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:13.361232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361089 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:13.361232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361157 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:13.361429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361247 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.361429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361261 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.361429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361270 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.361429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.361284 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.363321 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.363293 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:13.363460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.363320 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:13.363460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.363356 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "kube-api-access-bt9vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:13.363585 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.363485 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo" pod "55e32ea6-02e2-4808-ac8e-f79f2948cfeb" (UID: "55e32ea6-02e2-4808-ac8e-f79f2948cfeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:13.462839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.462804 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.463011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.462842 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.463011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.462859 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt9vm\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-kube-api-access-bt9vm\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.463011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.462873 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.463011 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.462888 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e32ea6-02e2-4808-ac8e-f79f2948cfeb-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:14:13.853444 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.853409 4494 generic.go:356] "Generic (PLEG): container finished" podID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerID="eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32" exitCode=0 Nov 03 00:14:13.853652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.853461 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" event={"ID":"55e32ea6-02e2-4808-ac8e-f79f2948cfeb","Type":"ContainerDied","Data":"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32"} Nov 03 00:14:13.853652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.853462 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" Nov 03 00:14:13.853652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.853502 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8" event={"ID":"55e32ea6-02e2-4808-ac8e-f79f2948cfeb","Type":"ContainerDied","Data":"5e4814d544021c3b52755b67b1f2fcd66021b110d3826b1f7cd353c0c1306a79"} Nov 03 00:14:13.853652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.853523 4494 scope.go:122] "RemoveContainer" containerID="eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32" Nov 03 00:14:13.862636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.862616 4494 scope.go:122] "RemoveContainer" containerID="eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32" Nov 03 00:14:13.862901 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:14:13.862883 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32\": container with ID starting with eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32 not found: ID does not exist" containerID="eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32" Nov 03 00:14:13.862955 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:13.862913 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32"} err="failed to get container status \"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32\": rpc error: code = NotFound desc = could not find container \"eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32\": container with ID starting with eed035701bf22f16a90edef02cc48d5668d6f38c4156ea8b93a18a0ba47d5b32 not found: ID does not exist" Nov 03 00:14:14.019911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:14.019836 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 03 00:14:14.070355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:14.070320 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5rkg8"] Nov 03 00:14:15.693838 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:15.693804 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" path="/var/lib/kubelet/pods/55e32ea6-02e2-4808-ac8e-f79f2948cfeb/volumes" Nov 03 00:14:17.558897 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:14:17.558801 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:14:18.054314 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:18.054276 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:18.200664 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:18.200629 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:19.292276 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:19.292245 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:19.438188 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:19.438131 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:19.753004 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:19.752974 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:19.895673 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:19.895635 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:23.260729 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:23.260697 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:23.900467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:23.900434 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:25.052794 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:25.052761 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:28.248592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:28.248558 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:30.056150 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:30.056108 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:33.149226 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:33.149162 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:37.428306 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:37.428275 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.152594 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:39.152561 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.677063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:39.677024 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:39.820815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:39.820774 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.052001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:44.051971 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.462861 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:44.462832 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:44.615073 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:44.615024 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:45.558561 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:45.558522 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:14:46.158345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:46.157876 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:46.305143 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:46.305109 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:47.168662 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:47.168625 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:47.560061 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:14:47.559973 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:14:49.058916 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:49.058882 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.357518 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:50.357482 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:50.859951 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:50.859909 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:52.707935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:52.707899 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:52.957809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:52.957766 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:54.338692 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:54.338658 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:14:59.251938 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:14:59.251897 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:04.160665 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:04.160617 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:09.157363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:09.157323 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:09.961075 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:09.961042 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:15:14.064825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:14.064794 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:15.264631 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.264592 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z"] Nov 03 00:15:15.265265 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265245 4494 cpu_manager.go:414] "containerMap: removing container" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265266 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265279 4494 cpu_manager.go:414] "containerMap: removing container" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265286 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265305 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265312 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265327 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2259f9bc-a646-45b1-9ba4-71ae5038d540" containerName="payload-processing" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265333 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2259f9bc-a646-45b1-9ba4-71ae5038d540" containerName="payload-processing" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265343 4494 cpu_manager.go:414] "containerMap: removing container" podUID="967731db-2140-4a0c-a9d3-2272b862aa67" containerName="maas-api" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265350 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967731db-2140-4a0c-a9d3-2272b862aa67" containerName="maas-api" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265365 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265371 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.265387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265389 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265396 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265406 4494 cpu_manager.go:414] "containerMap: removing container" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265413 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265428 4494 cpu_manager.go:414] "containerMap: removing container" podUID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" containerName="payload-pre-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265435 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" containerName="payload-pre-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265449 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265456 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265468 4494 cpu_manager.go:414] "containerMap: removing container" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265476 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265488 4494 cpu_manager.go:414] "containerMap: removing container" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265495 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265513 4494 cpu_manager.go:414] "containerMap: removing container" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265522 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265544 4494 cpu_manager.go:414] "containerMap: removing container" podUID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" containerName="maas-api" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265551 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" containerName="maas-api" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265563 4494 cpu_manager.go:414] "containerMap: removing container" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265570 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265593 4494 cpu_manager.go:414] "containerMap: removing container" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265600 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265705 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265715 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265727 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e43dbc-5e5d-44f5-bc84-6dd6f992a1c6" containerName="maas-api" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265738 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e32ea6-02e2-4808-ac8e-f79f2948cfeb" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265749 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265757 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265766 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="091c3f43-8afb-40cb-978c-fffd8b2bd867" containerName="payload-pre-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265775 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8414450-c908-4418-8983-2f2f9efe1695" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265785 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d840d5-3a79-4ef6-8214-0bce2361db7b" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265795 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2259f9bc-a646-45b1-9ba4-71ae5038d540" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265803 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="967731db-2140-4a0c-a9d3-2272b862aa67" containerName="maas-api" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265813 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3640416e-01aa-4410-b91e-8937ad196b86" containerName="istio-proxy" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265824 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a4b01-50be-41cb-9cff-2bf4ada7592d" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265834 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="726e5179-b6a5-48d5-a8c3-a570d90e97f2" containerName="payload-pre-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265844 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7815790-5819-412d-a30d-dd0541587c3b" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265855 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0e0549-0c5e-4a72-9814-46c74ac30bc2" containerName="payload-processing" Nov 03 00:15:15.266049 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.265862 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="159a248d-7ffa-4fa5-b2f6-11690ecac868" containerName="istio-proxy" Nov 03 00:15:15.270764 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.270744 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.325511 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.325467 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z"] Nov 03 00:15:15.373693 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.373663 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:15:15.373693 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.373665 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:15.429532 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.429501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m664h\" (UniqueName: \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.429701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.429648 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.429760 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.429724 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.530517 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.530432 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.530517 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.530480 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.530761 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.530545 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m664h\" (UniqueName: \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.531221 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.531195 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.532941 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.532920 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.592901 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.592864 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m664h\" (UniqueName: \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\") pod \"maas-api-revoke-sub-ed4f3be0-clj8z\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:15.883801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:15.883718 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:16.256853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:16.256832 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z"] Nov 03 00:15:16.258826 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:15:16.258791 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d621b40_5cf8_4260_b41b_76367c4bc501.slice/crio-55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223 WatchSource:0}: Error finding container 55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223: Status 404 returned error can't find the container with id 55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223 Nov 03 00:15:17.191828 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:17.191791 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" event={"ID":"2d621b40-5cf8-4260-b41b-76367c4bc501","Type":"ContainerStarted","Data":"dd43e194e6caa983a4436988a6efc2be7cf5ca256e7bc8b64cdf74fcf2016606"} Nov 03 00:15:17.192258 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:17.191843 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" event={"ID":"2d621b40-5cf8-4260-b41b-76367c4bc501","Type":"ContainerStarted","Data":"55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223"} Nov 03 00:15:17.309448 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:17.309392 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" podStartSLOduration=2.309377289 podStartE2EDuration="2.309377289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:17.308193142 +0000 UTC m=+5231.738062998" watchObservedRunningTime="2026-11-03 00:15:17.309377289 +0000 UTC m=+5231.739247150" Nov 03 00:15:17.559306 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:15:17.559213 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:15:18.199052 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:18.199021 4494 generic.go:356] "Generic (PLEG): container finished" podID="2d621b40-5cf8-4260-b41b-76367c4bc501" containerID="dd43e194e6caa983a4436988a6efc2be7cf5ca256e7bc8b64cdf74fcf2016606" exitCode=0 Nov 03 00:15:18.199496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:18.199112 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" event={"ID":"2d621b40-5cf8-4260-b41b-76367c4bc501","Type":"ContainerDied","Data":"dd43e194e6caa983a4436988a6efc2be7cf5ca256e7bc8b64cdf74fcf2016606"} Nov 03 00:15:18.955473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:18.955443 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:19.341963 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.341941 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:19.372405 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.372346 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\") pod \"2d621b40-5cf8-4260-b41b-76367c4bc501\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " Nov 03 00:15:19.372670 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.372433 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\") pod \"2d621b40-5cf8-4260-b41b-76367c4bc501\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " Nov 03 00:15:19.372670 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.372483 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\" (UniqueName: \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\") pod \"2d621b40-5cf8-4260-b41b-76367c4bc501\" (UID: \"2d621b40-5cf8-4260-b41b-76367c4bc501\") " Nov 03 00:15:19.372804 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.372666 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca" pod "2d621b40-5cf8-4260-b41b-76367c4bc501" (UID: "2d621b40-5cf8-4260-b41b-76367c4bc501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:19.372853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.372837 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.374774 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.374748 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca" pod "2d621b40-5cf8-4260-b41b-76367c4bc501" (UID: "2d621b40-5cf8-4260-b41b-76367c4bc501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:19.374874 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.374799 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h" pod "2d621b40-5cf8-4260-b41b-76367c4bc501" (UID: "2d621b40-5cf8-4260-b41b-76367c4bc501"). InnerVolumeSpecName "kube-api-access-m664h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:19.473896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.473855 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d621b40-5cf8-4260-b41b-76367c4bc501-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.473896 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:19.473885 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m664h\" (UniqueName: \"kubernetes.io/projected/2d621b40-5cf8-4260-b41b-76367c4bc501-kube-api-access-m664h\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.211102 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:20.211063 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" event={"ID":"2d621b40-5cf8-4260-b41b-76367c4bc501","Type":"ContainerDied","Data":"55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223"} Nov 03 00:15:20.211102 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:20.211084 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z" Nov 03 00:15:20.211347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:20.211139 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c028d9a56f00ba5b24aa636645829085ff24782c53df81cd4dc66cd77de223" Nov 03 00:15:23.576881 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:23.576851 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 03 00:15:23.577393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:23.577067 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" containerID="cri-o://f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616" gracePeriod=30 Nov 03 00:15:23.856398 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:23.856288 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:23.861342 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:23.861313 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6dlpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:15:23.861499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:23.861363 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:15:25.393465 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:25.393432 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:28.828101 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.828077 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 03 00:15:28.845468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.845437 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:28.871690 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871640 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871690 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871681 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871854 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871706 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871854 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871735 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871847 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871883 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871919 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.871977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871966 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.872246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871982 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:28.872246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.871997 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\") pod \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\" (UID: \"bd07ebce-72d2-4d4a-bc36-629c06d07fde\") " Nov 03 00:15:28.872246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872197 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:28.872246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872220 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:28.872497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872302 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:28.872614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872586 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:28.872799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872748 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.872799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872773 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.872799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872790 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.873000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872803 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.873000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.872818 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.874245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.874221 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:28.874245 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.874230 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:28.874384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.874257 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "kube-api-access-l2wq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:28.874384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.874332 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo" pod "bd07ebce-72d2-4d4a-bc36-629c06d07fde" (UID: "bd07ebce-72d2-4d4a-bc36-629c06d07fde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:15:28.973727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.973699 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.973727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.973725 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.974201 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.973736 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wq5\" (UniqueName: \"kubernetes.io/projected/bd07ebce-72d2-4d4a-bc36-629c06d07fde-kube-api-access-l2wq5\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.974201 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:28.973746 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd07ebce-72d2-4d4a-bc36-629c06d07fde-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:29.258551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.258520 4494 generic.go:356] "Generic (PLEG): container finished" podID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerID="f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616" exitCode=0 Nov 03 00:15:29.258771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.258557 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" event={"ID":"bd07ebce-72d2-4d4a-bc36-629c06d07fde","Type":"ContainerDied","Data":"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616"} Nov 03 00:15:29.258771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.258570 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" Nov 03 00:15:29.258771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.258597 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr" event={"ID":"bd07ebce-72d2-4d4a-bc36-629c06d07fde","Type":"ContainerDied","Data":"d8ba85fb2afde49b7bace8a3c3c081be6405ac682f1abf91db73ad61b108f6cf"} Nov 03 00:15:29.258771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.258618 4494 scope.go:122] "RemoveContainer" containerID="f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616" Nov 03 00:15:29.274763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.274730 4494 scope.go:122] "RemoveContainer" containerID="f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616" Nov 03 00:15:29.275077 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:15:29.275053 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616\": container with ID starting with f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616 not found: ID does not exist" containerID="f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616" Nov 03 00:15:29.275157 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.275089 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616"} err="failed to get container status \"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616\": rpc error: code = NotFound desc = could not find container \"f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616\": container with ID starting with f7ecc99959c6c1ce5390eb08c5475807265cd7fbe942095521e4cca21f12b616 not found: ID does not exist" Nov 03 00:15:29.425998 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.425969 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 03 00:15:29.431027 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.430998 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6dlpr"] Nov 03 00:15:29.692371 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:29.692341 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" path="/var/lib/kubelet/pods/bd07ebce-72d2-4d4a-bc36-629c06d07fde/volumes" Nov 03 00:15:30.521568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:30.521533 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:15:33.753820 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:33.753787 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:34.244921 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:34.244889 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:34.389730 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:34.389701 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:38.653797 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:38.653764 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:43.540153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:43.540111 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:44.739696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:44.739665 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:15:47.561235 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:15:47.561139 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:15:47.827737 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:47.827613 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:47.971421 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:47.971375 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:48.556525 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:48.556495 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:53.448736 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:53.448700 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:15:58.362323 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:15:58.362291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:00.470097 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:00.469988 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:03.253082 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:03.253047 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:08.268429 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:08.268390 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:11.277951 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:11.277885 4494 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal" podUID="b8bcc47f51109dce7950aafef82fe134" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 03 00:16:13.151979 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:13.151945 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:15.259530 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:15.259484 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:15.725383 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:15.725341 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:17.559988 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:17.559949 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:16:18.139010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:18.138972 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.052887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:22.052847 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:22.248024 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:22.247985 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:23.136966 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:23.136927 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:23.283628 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:23.283584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:25.736841 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:25.736804 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:27.979133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:27.979101 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:28.368834 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:28.368738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:33.341490 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:33.341455 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:34.000128 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:34.000092 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:16:37.526024 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.525984 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:16:37.526483 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.526193 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" podUID="9b10f152-5990-468a-918b-3c6b6ed63e35" containerName="maas-api" containerID="cri-o://53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b" gracePeriod=30 Nov 03 00:16:37.634128 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.634092 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:37.787532 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.787471 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:16:37.878405 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.878377 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\") pod \"9b10f152-5990-468a-918b-3c6b6ed63e35\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " Nov 03 00:16:37.878622 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.878506 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\" (UniqueName: \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\") pod \"9b10f152-5990-468a-918b-3c6b6ed63e35\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " Nov 03 00:16:37.878622 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.878538 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") pod \"9b10f152-5990-468a-918b-3c6b6ed63e35\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " Nov 03 00:16:37.878622 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.878586 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\") pod \"9b10f152-5990-468a-918b-3c6b6ed63e35\" (UID: \"9b10f152-5990-468a-918b-3c6b6ed63e35\") " Nov 03 00:16:37.878986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.878958 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca" pod "9b10f152-5990-468a-918b-3c6b6ed63e35" (UID: "9b10f152-5990-468a-918b-3c6b6ed63e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:37.880765 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.880732 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls" pod "9b10f152-5990-468a-918b-3c6b6ed63e35" (UID: "9b10f152-5990-468a-918b-3c6b6ed63e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:37.880875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.880805 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs" pod "9b10f152-5990-468a-918b-3c6b6ed63e35" (UID: "9b10f152-5990-468a-918b-3c6b6ed63e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:37.880875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.880807 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6" pod "9b10f152-5990-468a-918b-3c6b6ed63e35" (UID: "9b10f152-5990-468a-918b-3c6b6ed63e35"). InnerVolumeSpecName "kube-api-access-gx5t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:37.980053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.980018 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5t6\" (UniqueName: \"kubernetes.io/projected/9b10f152-5990-468a-918b-3c6b6ed63e35-kube-api-access-gx5t6\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.980053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.980051 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.980053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.980061 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b10f152-5990-468a-918b-3c6b6ed63e35-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.980347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:37.980069 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b10f152-5990-468a-918b-3c6b6ed63e35-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.634648 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.634611 4494 generic.go:356] "Generic (PLEG): container finished" podID="9b10f152-5990-468a-918b-3c6b6ed63e35" containerID="53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b" exitCode=0 Nov 03 00:16:38.635087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.634662 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" Nov 03 00:16:38.635087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.634695 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" event={"ID":"9b10f152-5990-468a-918b-3c6b6ed63e35","Type":"ContainerDied","Data":"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b"} Nov 03 00:16:38.635087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.634756 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb" event={"ID":"9b10f152-5990-468a-918b-3c6b6ed63e35","Type":"ContainerDied","Data":"0689a5bfd2187a4c1a9d3c352a0b8d581a6d0d26e2ab3f5a01b925cc8e84c612"} Nov 03 00:16:38.635087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.634785 4494 scope.go:122] "RemoveContainer" containerID="53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b" Nov 03 00:16:38.645010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.644986 4494 scope.go:122] "RemoveContainer" containerID="53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b" Nov 03 00:16:38.645337 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:38.645314 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b\": container with ID starting with 53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b not found: ID does not exist" containerID="53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b" Nov 03 00:16:38.645422 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.645347 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b"} err="failed to get container status \"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b\": rpc error: code = NotFound desc = could not find container \"53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b\": container with ID starting with 53a3ace52e5436b09cb50df3449cddb417076c5032022aa6181d35e27ee8b09b not found: ID does not exist" Nov 03 00:16:38.804327 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.804298 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:16:38.809183 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:38.809153 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bee1b-6b8f9c8449-x5bqb"] Nov 03 00:16:39.350646 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:39.350617 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:39.700589 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:39.700553 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b10f152-5990-468a-918b-3c6b6ed63e35" path="/var/lib/kubelet/pods/9b10f152-5990-468a-918b-3c6b6ed63e35/volumes" Nov 03 00:16:42.412831 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:42.412798 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:42.558275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:42.558237 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:43.965394 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:43.965362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:44.073036 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:44.073005 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:16:44.527961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:44.527885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:44.590100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:44.590067 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:16:44.590359 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:44.590333 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" containerID="cri-o://9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24" gracePeriod=30 Nov 03 00:16:44.682616 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:44.682585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:45.368351 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:45.368318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:47.559158 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:47.559123 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:16:49.658637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:49.658601 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:49.841284 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:49.841261 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:16:50.004579 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004531 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004579 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004581 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004655 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004683 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004741 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004780 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.004819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004806 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.005046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004840 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.005046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004847 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:50.005046 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.004881 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\") pod \"0d294e2f-18e9-478a-99cc-8a307195ad58\" (UID: \"0d294e2f-18e9-478a-99cc-8a307195ad58\") " Nov 03 00:16:50.005310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005120 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:50.005310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005164 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:50.005310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005280 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:50.005476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005328 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.005476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005349 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.005476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.005361 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:50.007267 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.007239 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:50.007392 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.007294 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:50.007505 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.007487 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:50.007553 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.007515 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49" pod "0d294e2f-18e9-478a-99cc-8a307195ad58" (UID: "0d294e2f-18e9-478a-99cc-8a307195ad58"). InnerVolumeSpecName "kube-api-access-6bf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:50.106447 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106415 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106447 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106444 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bf49\" (UniqueName: \"kubernetes.io/projected/0d294e2f-18e9-478a-99cc-8a307195ad58-kube-api-access-6bf49\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106447 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106455 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d294e2f-18e9-478a-99cc-8a307195ad58-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106463 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106471 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106481 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.106722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.106489 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d294e2f-18e9-478a-99cc-8a307195ad58-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:50.697414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.697381 4494 generic.go:356] "Generic (PLEG): container finished" podID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerID="9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24" exitCode=0 Nov 03 00:16:50.697414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.697425 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerDied","Data":"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24"} Nov 03 00:16:50.697883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.697471 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" event={"ID":"0d294e2f-18e9-478a-99cc-8a307195ad58","Type":"ContainerDied","Data":"1392d1239cb46bfd3ede1782454b9dae9a3f33b528956b213730f63bde3ba8a4"} Nov 03 00:16:50.697883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.697495 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8" Nov 03 00:16:50.697883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.697494 4494 scope.go:122] "RemoveContainer" containerID="9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24" Nov 03 00:16:50.707487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.707466 4494 scope.go:122] "RemoveContainer" containerID="fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819" Nov 03 00:16:50.716434 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.716286 4494 scope.go:122] "RemoveContainer" containerID="9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24" Nov 03 00:16:50.716548 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:50.716526 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24\": container with ID starting with 9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24 not found: ID does not exist" containerID="9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24" Nov 03 00:16:50.716604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.716556 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24"} err="failed to get container status \"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24\": rpc error: code = NotFound desc = could not find container \"9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24\": container with ID starting with 9269c5737d239cf286ef828dcbfd73cab1a1089ecb5ccab629a9d14bc006dc24 not found: ID does not exist" Nov 03 00:16:50.716604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.716576 4494 scope.go:122] "RemoveContainer" containerID="fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819" Nov 03 00:16:50.716813 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:50.716796 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819\": container with ID starting with fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819 not found: ID does not exist" containerID="fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819" Nov 03 00:16:50.716857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.716819 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819"} err="failed to get container status \"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819\": rpc error: code = NotFound desc = could not find container \"fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819\": container with ID starting with fce0a670d74307c583f0e3d37b6380d19299f4e80d57dcb9708c29b5b53e7819 not found: ID does not exist" Nov 03 00:16:50.915457 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.915422 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:16:50.965614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:50.965517 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bee1b-openshift-default-86498d7c65-5h2z8"] Nov 03 00:16:51.603311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:51.603262 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:51.692897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:51.692863 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" path="/var/lib/kubelet/pods/0d294e2f-18e9-478a-99cc-8a307195ad58/volumes" Nov 03 00:16:52.226653 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:52.226614 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:52.369836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:52.369805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:54.831384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:54.831354 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:57.132723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:57.132687 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:59.132555 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.132518 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:16:59.133004 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.132739 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" podUID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" containerName="maas-api" containerID="cri-o://d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5" gracePeriod=30 Nov 03 00:16:59.276508 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.276451 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:16:59.391075 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.391011 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:16:59.401329 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.401303 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\") pod \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " Nov 03 00:16:59.401468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.401341 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\" (UniqueName: \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\") pod \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " Nov 03 00:16:59.401468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.401365 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") pod \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " Nov 03 00:16:59.401736 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.401708 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca" pod "b2a85a16-3bbf-4607-aaaa-ff32cee2c595" (UID: "b2a85a16-3bbf-4607-aaaa-ff32cee2c595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:59.403511 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.403488 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck" pod "b2a85a16-3bbf-4607-aaaa-ff32cee2c595" (UID: "b2a85a16-3bbf-4607-aaaa-ff32cee2c595"). InnerVolumeSpecName "kube-api-access-nt5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:59.403511 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.403497 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls" pod "b2a85a16-3bbf-4607-aaaa-ff32cee2c595" (UID: "b2a85a16-3bbf-4607-aaaa-ff32cee2c595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:59.502578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.502548 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\") pod \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\" (UID: \"b2a85a16-3bbf-4607-aaaa-ff32cee2c595\") " Nov 03 00:16:59.502762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.502749 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:59.502817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.502764 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5ck\" (UniqueName: \"kubernetes.io/projected/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-kube-api-access-nt5ck\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:59.502817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.502776 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:59.504709 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.504675 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs" pod "b2a85a16-3bbf-4607-aaaa-ff32cee2c595" (UID: "b2a85a16-3bbf-4607-aaaa-ff32cee2c595"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:59.603345 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.603310 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a85a16-3bbf-4607-aaaa-ff32cee2c595-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:59.749546 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.749516 4494 generic.go:356] "Generic (PLEG): container finished" podID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" containerID="d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5" exitCode=0 Nov 03 00:16:59.749695 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.749571 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" Nov 03 00:16:59.749695 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.749603 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" event={"ID":"b2a85a16-3bbf-4607-aaaa-ff32cee2c595","Type":"ContainerDied","Data":"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5"} Nov 03 00:16:59.749695 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.749659 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx" event={"ID":"b2a85a16-3bbf-4607-aaaa-ff32cee2c595","Type":"ContainerDied","Data":"ef6e4098b8ad34ae867a539e3b762cc43f7fa9a41f0765175dbb4864cc7c2778"} Nov 03 00:16:59.749695 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.749689 4494 scope.go:122] "RemoveContainer" containerID="d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5" Nov 03 00:16:59.758868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.758851 4494 scope.go:122] "RemoveContainer" containerID="d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5" Nov 03 00:16:59.759134 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:16:59.759116 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5\": container with ID starting with d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5 not found: ID does not exist" containerID="d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5" Nov 03 00:16:59.759216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.759144 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5"} err="failed to get container status \"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5\": rpc error: code = NotFound desc = could not find container \"d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5\": container with ID starting with d001da07997313cf0cb51d0a690696d9c43e0a8b6a04a10ace9d301973834ad5 not found: ID does not exist" Nov 03 00:16:59.916710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.916676 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:16:59.967029 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:16:59.966947 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9b6c9e-7f6b9b8bd6-hnmxx"] Nov 03 00:17:01.041750 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:01.041672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:01.692597 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:01.692563 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" path="/var/lib/kubelet/pods/b2a85a16-3bbf-4607-aaaa-ff32cee2c595/volumes" Nov 03 00:17:03.000283 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:03.000246 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:03.146635 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:03.146603 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:04.545973 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:04.545939 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:04.790374 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:04.790333 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:17:05.030496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:05.030466 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:17:05.030746 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:05.030717 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" containerID="cri-o://b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f" gracePeriod=30 Nov 03 00:17:05.042927 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:05.042887 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:05.496059 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:05.496012 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:06.125409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125372 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf"] Nov 03 00:17:06.125821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125811 4494 cpu_manager.go:414] "containerMap: removing container" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" Nov 03 00:17:06.125870 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125823 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" Nov 03 00:17:06.125870 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125846 4494 cpu_manager.go:414] "containerMap: removing container" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:17:06.125870 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125851 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48face08-0636-4c00-b7e8-9e8de6ce7983" containerName="istio-proxy" Nov 03 00:17:06.125870 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125865 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125873 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125881 4494 cpu_manager.go:414] "containerMap: removing container" podUID="9b10f152-5990-468a-918b-3c6b6ed63e35" containerName="maas-api" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125886 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b10f152-5990-468a-918b-3c6b6ed63e35" containerName="maas-api" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125904 4494 cpu_manager.go:414] "containerMap: removing container" podUID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" containerName="maas-api" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125910 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" containerName="maas-api" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125916 4494 cpu_manager.go:414] "containerMap: removing container" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125920 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125930 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2d621b40-5cf8-4260-b41b-76367c4bc501" containerName="revoke-keys" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.125933 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d621b40-5cf8-4260-b41b-76367c4bc501" containerName="revoke-keys" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126006 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8426ee-2058-4c4a-b96d-c503316a01bf" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126020 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126029 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b10f152-5990-468a-918b-3c6b6ed63e35" containerName="maas-api" Nov 03 00:17:06.126451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126039 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a85a16-3bbf-4607-aaaa-ff32cee2c595" containerName="maas-api" Nov 03 00:17:06.126451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126045 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d294e2f-18e9-478a-99cc-8a307195ad58" containerName="istio-proxy" Nov 03 00:17:06.126451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126050 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d621b40-5cf8-4260-b41b-76367c4bc501" containerName="revoke-keys" Nov 03 00:17:06.126451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.126059 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd07ebce-72d2-4d4a-bc36-629c06d07fde" containerName="istio-proxy" Nov 03 00:17:06.130773 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.130751 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.131583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.131556 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:06.165588 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.165556 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhvb\" (UniqueName: \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.165771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.165623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.165771 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.165729 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.186452 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.186417 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf"] Nov 03 00:17:06.234611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.234574 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:06.234787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.234654 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:17:06.266841 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.266805 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhvb\" (UniqueName: \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.267048 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.266910 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.267048 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.266978 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.267698 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.267670 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.269567 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.269548 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.328969 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.328933 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhvb\" (UniqueName: \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\") pod \"maas-api-revoke-sub-36ec64ed-nfqnf\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.444103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.444067 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:06.817750 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:06.817727 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf"] Nov 03 00:17:06.819535 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:17:06.819507 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40ccabe_7e53_421a_984b_89f13b45994e.slice/crio-7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72 WatchSource:0}: Error finding container 7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72: Status 404 returned error can't find the container with id 7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72 Nov 03 00:17:07.793057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:07.793011 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" event={"ID":"f40ccabe-7e53-421a-984b-89f13b45994e","Type":"ContainerStarted","Data":"99a047ee371557ad893365a3827deeb9425b28395f90b964c4bc4012fab1225c"} Nov 03 00:17:07.793556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:07.793088 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" event={"ID":"f40ccabe-7e53-421a-984b-89f13b45994e","Type":"ContainerStarted","Data":"7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72"} Nov 03 00:17:07.910240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:07.910158 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" podStartSLOduration=1.910142906 podStartE2EDuration="1.910142906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:07.909596947 +0000 UTC m=+5342.339466810" watchObservedRunningTime="2026-11-03 00:17:07.910142906 +0000 UTC m=+5342.340012767" Nov 03 00:17:08.799855 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:08.799822 4494 generic.go:356] "Generic (PLEG): container finished" podID="f40ccabe-7e53-421a-984b-89f13b45994e" containerID="99a047ee371557ad893365a3827deeb9425b28395f90b964c4bc4012fab1225c" exitCode=0 Nov 03 00:17:08.800313 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:08.799908 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" event={"ID":"f40ccabe-7e53-421a-984b-89f13b45994e","Type":"ContainerDied","Data":"99a047ee371557ad893365a3827deeb9425b28395f90b964c4bc4012fab1225c"} Nov 03 00:17:09.932561 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:09.932537 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:10.005304 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.005269 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\") pod \"f40ccabe-7e53-421a-984b-89f13b45994e\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " Nov 03 00:17:10.005470 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.005332 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\" (UniqueName: \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\") pod \"f40ccabe-7e53-421a-984b-89f13b45994e\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " Nov 03 00:17:10.005470 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.005394 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\") pod \"f40ccabe-7e53-421a-984b-89f13b45994e\" (UID: \"f40ccabe-7e53-421a-984b-89f13b45994e\") " Nov 03 00:17:10.005734 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.005702 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca" pod "f40ccabe-7e53-421a-984b-89f13b45994e" (UID: "f40ccabe-7e53-421a-984b-89f13b45994e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:10.007656 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.007631 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca" pod "f40ccabe-7e53-421a-984b-89f13b45994e" (UID: "f40ccabe-7e53-421a-984b-89f13b45994e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:10.007768 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.007749 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb" pod "f40ccabe-7e53-421a-984b-89f13b45994e" (UID: "f40ccabe-7e53-421a-984b-89f13b45994e"). InnerVolumeSpecName "kube-api-access-jjhvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:10.106787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.106746 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.106787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.106766 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhvb\" (UniqueName: \"kubernetes.io/projected/f40ccabe-7e53-421a-984b-89f13b45994e-kube-api-access-jjhvb\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.106787 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.106776 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40ccabe-7e53-421a-984b-89f13b45994e-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.265572 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.265548 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:17:10.308301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308270 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308310 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308363 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308393 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308431 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308458 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308591 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:10.308753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308634 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308673 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308710 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\") pod \"d4ac90de-2850-48b9-bb53-f4f73c6911af\" (UID: \"d4ac90de-2850-48b9-bb53-f4f73c6911af\") " Nov 03 00:17:10.308982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308772 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:10.308982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308905 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:10.308982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.308963 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:10.309167 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309090 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:10.309428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309205 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.309428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309223 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4ac90de-2850-48b9-bb53-f4f73c6911af-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.309428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309235 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.309428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309244 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.309428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.309252 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.310990 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.310929 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:10.310990 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.310948 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:10.311207 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.311190 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "kube-api-access-xjtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:10.311274 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.311201 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy" pod "d4ac90de-2850-48b9-bb53-f4f73c6911af" (UID: "d4ac90de-2850-48b9-bb53-f4f73c6911af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:10.409968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.409935 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.409968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.409963 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.409968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.409973 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtww\" (UniqueName: \"kubernetes.io/projected/d4ac90de-2850-48b9-bb53-f4f73c6911af-kube-api-access-xjtww\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.410247 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.409984 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4ac90de-2850-48b9-bb53-f4f73c6911af-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:10.553948 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.553912 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:10.811648 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.811613 4494 generic.go:356] "Generic (PLEG): container finished" podID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerID="b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f" exitCode=0 Nov 03 00:17:10.811868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.811674 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" Nov 03 00:17:10.811868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.811707 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerDied","Data":"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f"} Nov 03 00:17:10.811868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.811775 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2" event={"ID":"d4ac90de-2850-48b9-bb53-f4f73c6911af","Type":"ContainerDied","Data":"aadd36dc5305db6eca9927caaf9101c8f1d05a90e90eba7a3c440e321c9d1345"} Nov 03 00:17:10.811868 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.811806 4494 scope.go:122] "RemoveContainer" containerID="b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f" Nov 03 00:17:10.813260 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.813146 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" event={"ID":"f40ccabe-7e53-421a-984b-89f13b45994e","Type":"ContainerDied","Data":"7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72"} Nov 03 00:17:10.813260 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.813195 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf" Nov 03 00:17:10.813397 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.813197 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dad1c9177c44904de6fdfac82518d9acf915fcb16eb75587651b29611898a72" Nov 03 00:17:10.821703 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.821687 4494 scope.go:122] "RemoveContainer" containerID="0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb" Nov 03 00:17:10.832465 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.832444 4494 scope.go:122] "RemoveContainer" containerID="b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f" Nov 03 00:17:10.832720 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:17:10.832699 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f\": container with ID starting with b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f not found: ID does not exist" containerID="b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f" Nov 03 00:17:10.832795 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.832733 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f"} err="failed to get container status \"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f\": rpc error: code = NotFound desc = could not find container \"b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f\": container with ID starting with b5e17aa1c5989b082addbe725e06697e3e756d501af01e65970c2622fe05e50f not found: ID does not exist" Nov 03 00:17:10.832795 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.832757 4494 scope.go:122] "RemoveContainer" containerID="0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb" Nov 03 00:17:10.833024 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:17:10.833008 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb\": container with ID starting with 0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb not found: ID does not exist" containerID="0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb" Nov 03 00:17:10.833080 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:10.833029 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb"} err="failed to get container status \"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb\": rpc error: code = NotFound desc = could not find container \"0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb\": container with ID starting with 0e3ec4c507b5a5acd64b89ab3580795ee3ec56ae0b537e23f451b5b1ca0604fb not found: ID does not exist" Nov 03 00:17:11.031754 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:11.031718 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:17:11.037352 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:11.037325 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9b6c9e-openshift-default-b897bc855-vw5k2"] Nov 03 00:17:11.249375 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:11.249338 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:11.691994 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:11.691963 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" path="/var/lib/kubelet/pods/d4ac90de-2850-48b9-bb53-f4f73c6911af/volumes" Nov 03 00:17:12.471266 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:12.471200 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:16.143886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:16.143849 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:16.567931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:16.567893 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:17:17.560578 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:17:17.560466 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:17:18.446998 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.446966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:18.751864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.751778 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v"] Nov 03 00:17:18.752469 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752447 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752471 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752490 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752497 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752519 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f40ccabe-7e53-421a-984b-89f13b45994e" containerName="revoke-keys" Nov 03 00:17:18.752565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752526 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40ccabe-7e53-421a-984b-89f13b45994e" containerName="revoke-keys" Nov 03 00:17:18.752847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752645 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752659 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ac90de-2850-48b9-bb53-f4f73c6911af" containerName="istio-proxy" Nov 03 00:17:18.752847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.752677 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40ccabe-7e53-421a-984b-89f13b45994e" containerName="revoke-keys" Nov 03 00:17:18.757503 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.757481 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.810669 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.810640 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:18.813331 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.813308 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v"] Nov 03 00:17:18.861108 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.861079 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rx8wr\"" Nov 03 00:17:18.892107 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.892069 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.892317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.892123 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.892317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.892286 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx62p\" (UniqueName: \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.993682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.993639 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.993912 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.993700 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx62p\" (UniqueName: \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.993912 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.993819 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.994373 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.994348 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:18.996293 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:18.996273 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:19.055676 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.055598 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx62p\" (UniqueName: \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\") pod \"maas-api-revoke-sub-4bcfa411-4n97v\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:19.069892 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.069861 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:19.324489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.324351 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 03 00:17:19.443510 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.443434 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v"] Nov 03 00:17:19.445697 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:17:19.445671 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f9f778_17ff_43e2_a14e_68fd8914d1ee.slice/crio-4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507 WatchSource:0}: Error finding container 4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507: Status 404 returned error can't find the container with id 4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507 Nov 03 00:17:19.579723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.579644 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:19.728523 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.728489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:19.859995 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.859972 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" event={"ID":"11f9f778-17ff-43e2-a14e-68fd8914d1ee","Type":"ContainerStarted","Data":"39b9fdf943ec897d7908f418ec4a1a869e7d060bef9aea4728609ef50ddfb63c"} Nov 03 00:17:19.860351 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.860021 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" event={"ID":"11f9f778-17ff-43e2-a14e-68fd8914d1ee","Type":"ContainerStarted","Data":"4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507"} Nov 03 00:17:19.978089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:19.978033 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" podStartSLOduration=1.978015192 podStartE2EDuration="1.978015192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:19.976151651 +0000 UTC m=+5354.406021514" watchObservedRunningTime="2026-11-03 00:17:19.978015192 +0000 UTC m=+5354.407885052" Nov 03 00:17:20.866573 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:20.866478 4494 generic.go:356] "Generic (PLEG): container finished" podID="11f9f778-17ff-43e2-a14e-68fd8914d1ee" containerID="39b9fdf943ec897d7908f418ec4a1a869e7d060bef9aea4728609ef50ddfb63c" exitCode=0 Nov 03 00:17:20.866573 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:20.866552 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" event={"ID":"11f9f778-17ff-43e2-a14e-68fd8914d1ee","Type":"ContainerDied","Data":"39b9fdf943ec897d7908f418ec4a1a869e7d060bef9aea4728609ef50ddfb63c"} Nov 03 00:17:22.006616 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.006591 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:22.125973 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.125863 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\") pod \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " Nov 03 00:17:22.125973 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.125937 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\") pod \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " Nov 03 00:17:22.126248 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.125993 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\" (UniqueName: \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\") pod \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\" (UID: \"11f9f778-17ff-43e2-a14e-68fd8914d1ee\") " Nov 03 00:17:22.126562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.126491 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca" pod "11f9f778-17ff-43e2-a14e-68fd8914d1ee" (UID: "11f9f778-17ff-43e2-a14e-68fd8914d1ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:22.128865 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.128835 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p" pod "11f9f778-17ff-43e2-a14e-68fd8914d1ee" (UID: "11f9f778-17ff-43e2-a14e-68fd8914d1ee"). InnerVolumeSpecName "kube-api-access-gx62p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:22.129417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.129394 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca" pod "11f9f778-17ff-43e2-a14e-68fd8914d1ee" (UID: "11f9f778-17ff-43e2-a14e-68fd8914d1ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:22.227301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.227250 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:22.227301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.227293 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f9f778-17ff-43e2-a14e-68fd8914d1ee-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:22.227301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.227304 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx62p\" (UniqueName: \"kubernetes.io/projected/11f9f778-17ff-43e2-a14e-68fd8914d1ee-kube-api-access-gx62p\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:22.878515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.878472 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" event={"ID":"11f9f778-17ff-43e2-a14e-68fd8914d1ee","Type":"ContainerDied","Data":"4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507"} Nov 03 00:17:22.878707 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.878531 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bb8744e4b87a5605cfd79a8e8c00a7822056492d382c046846b936d18362507" Nov 03 00:17:22.878707 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:22.878497 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v" Nov 03 00:17:24.325932 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:24.325863 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" containerID="cri-o://a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b" gracePeriod=25 Nov 03 00:17:24.473799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:24.473759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:26.089378 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:26.089346 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:26.803560 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:26.803526 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:30.026587 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:30.026555 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:30.771627 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:30.771594 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:35.059961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:35.059868 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:17:35.642864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:35.642829 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:35.789883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:35.789855 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:36.804377 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:36.804342 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:44.576507 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.576482 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 03 00:17:44.648801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.648722 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\") pod \"44f343c1-f72a-4950-9434-5a3dcd238c1d\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " Nov 03 00:17:44.648801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.648763 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\" (UniqueName: \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\") pod \"44f343c1-f72a-4950-9434-5a3dcd238c1d\" (UID: \"44f343c1-f72a-4950-9434-5a3dcd238c1d\") " Nov 03 00:17:44.649143 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.649119 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume" pod "44f343c1-f72a-4950-9434-5a3dcd238c1d" (UID: "44f343c1-f72a-4950-9434-5a3dcd238c1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:44.650951 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.650933 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg" pod "44f343c1-f72a-4950-9434-5a3dcd238c1d" (UID: "44f343c1-f72a-4950-9434-5a3dcd238c1d"). InnerVolumeSpecName "kube-api-access-lb8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.750533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.750497 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f343c1-f72a-4950-9434-5a3dcd238c1d-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.750533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:44.750522 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8fg\" (UniqueName: \"kubernetes.io/projected/44f343c1-f72a-4950-9434-5a3dcd238c1d-kube-api-access-lb8fg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:45.004656 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.004611 4494 generic.go:356] "Generic (PLEG): container finished" podID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerID="a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b" exitCode=0 Nov 03 00:17:45.004847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.004670 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" Nov 03 00:17:45.004847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.004699 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" event={"ID":"44f343c1-f72a-4950-9434-5a3dcd238c1d","Type":"ContainerDied","Data":"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b"} Nov 03 00:17:45.004941 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.004884 4494 scope.go:122] "RemoveContainer" containerID="a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b" Nov 03 00:17:45.004941 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.004918 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc" event={"ID":"44f343c1-f72a-4950-9434-5a3dcd238c1d","Type":"ContainerDied","Data":"fdbd04d08645a8924d6c6591a45134bcf4098e3af40303a9a3e1c2d935f34381"} Nov 03 00:17:45.013958 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.013941 4494 scope.go:122] "RemoveContainer" containerID="a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b" Nov 03 00:17:45.014277 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:17:45.014257 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b\": container with ID starting with a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b not found: ID does not exist" containerID="a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b" Nov 03 00:17:45.014371 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.014285 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b"} err="failed to get container status \"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b\": rpc error: code = NotFound desc = could not find container \"a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b\": container with ID starting with a50a4c98d6227d730838c3fe3419e995850d115a5d4e2f04ac62fcfdfbc6bc7b not found: ID does not exist" Nov 03 00:17:45.171652 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.171619 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 03 00:17:45.175667 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.175643 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b42983-85f569d9ff-rkhpc"] Nov 03 00:17:45.692228 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:45.692196 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" path="/var/lib/kubelet/pods/44f343c1-f72a-4950-9434-5a3dcd238c1d/volumes" Nov 03 00:17:46.093874 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:46.093791 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:46.748150 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:46.748031 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:46.899275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:46.899237 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:47.563455 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:17:47.563419 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:17:47.949762 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:47.949728 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:52.465921 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:52.465868 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:52.924390 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:52.924353 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 03 00:17:52.924626 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:52.924591 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" containerID="cri-o://a33026425f628997739b68f9d7bc6180d867e709fdfaff3556ec585d02336796" gracePeriod=30 Nov 03 00:17:52.932922 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:52.932865 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:53.844501 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:53.844456 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:53.989812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:53.989774 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:17:56.338942 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:56.338913 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8dkp_5d0a16e7-cd0c-4161-80ea-cdb46ae08b91/limitador/0.log" Nov 03 00:17:58.072141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.072098 4494 generic.go:356] "Generic (PLEG): container finished" podID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerID="a33026425f628997739b68f9d7bc6180d867e709fdfaff3556ec585d02336796" exitCode=0 Nov 03 00:17:58.072636 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.072156 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" event={"ID":"d07b85b5-7585-40ac-bd0a-feeb294cafa5","Type":"ContainerDied","Data":"a33026425f628997739b68f9d7bc6180d867e709fdfaff3556ec585d02336796"} Nov 03 00:17:58.176231 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.176206 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 03 00:17:58.283310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283225 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283277 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283308 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283575 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283484 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283575 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283553 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283697 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283590 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.283697 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283599 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.283697 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283627 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283872 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283713 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283872 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283762 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.283872 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283776 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.283872 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283804 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\") pod \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\" (UID: \"d07b85b5-7585-40ac-bd0a-feeb294cafa5\") " Nov 03 00:17:58.284013 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.283970 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.284078 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284003 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:58.284274 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284245 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.284274 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284267 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.284480 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284280 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.284480 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284294 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.284480 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.284307 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.285735 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.285715 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:58.286000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.285984 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "kube-api-access-m6klg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:58.286393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.286372 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:58.286393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.286374 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy" pod "d07b85b5-7585-40ac-bd0a-feeb294cafa5" (UID: "d07b85b5-7585-40ac-bd0a-feeb294cafa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:58.385655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.385617 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6klg\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-kube-api-access-m6klg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.385655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.385648 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.385655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.385658 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:58.385911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:58.385666 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07b85b5-7585-40ac-bd0a-feeb294cafa5-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:17:59.078381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.078353 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" Nov 03 00:17:59.078381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.078364 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x" event={"ID":"d07b85b5-7585-40ac-bd0a-feeb294cafa5","Type":"ContainerDied","Data":"bf91a9ad48ba84270031aed90dc868af6974aec3b9d123ce3a6b5fef6ba7c1e9"} Nov 03 00:17:59.079040 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.078425 4494 scope.go:122] "RemoveContainer" containerID="a33026425f628997739b68f9d7bc6180d867e709fdfaff3556ec585d02336796" Nov 03 00:17:59.296489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.296454 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 03 00:17:59.346917 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.346841 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42983-openshift-default-c98888847-4nz8x"] Nov 03 00:17:59.473985 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.473950 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-nzlvl_d98647bf-90df-4e01-8959-b857f423b272/manager/1.log" Nov 03 00:17:59.691740 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:17:59.691708 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" path="/var/lib/kubelet/pods/d07b85b5-7585-40ac-bd0a-feeb294cafa5/volumes" Nov 03 00:18:00.846417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:00.846389 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-nzlvl_d98647bf-90df-4e01-8959-b857f423b272/manager/0.log" Nov 03 00:18:04.855839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:04.855812 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:05.361405 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361371 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:18:05.361796 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361783 4494 cpu_manager.go:414] "containerMap: removing container" podUID="11f9f778-17ff-43e2-a14e-68fd8914d1ee" containerName="revoke-keys" Nov 03 00:18:05.361859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361797 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f9f778-17ff-43e2-a14e-68fd8914d1ee" containerName="revoke-keys" Nov 03 00:18:05.361859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361818 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" Nov 03 00:18:05.361859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361823 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" Nov 03 00:18:05.361859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361842 4494 cpu_manager.go:414] "containerMap: removing container" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" Nov 03 00:18:05.361859 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361847 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" Nov 03 00:18:05.362033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361917 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f9f778-17ff-43e2-a14e-68fd8914d1ee" containerName="revoke-keys" Nov 03 00:18:05.362033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361928 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07b85b5-7585-40ac-bd0a-feeb294cafa5" containerName="istio-proxy" Nov 03 00:18:05.362033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.361938 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f343c1-f72a-4950-9434-5a3dcd238c1d" containerName="payload-pre-processing" Nov 03 00:18:05.366411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.366389 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.419562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.419518 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-018f8381-openshift-default-dockercfg-rqd6v\"" Nov 03 00:18:05.454346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454315 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454527 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454352 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454527 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454441 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gln69\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454527 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454493 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454726 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454594 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454726 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454637 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454726 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454688 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454771 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.454836 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.454816 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.472611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.472585 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:18:05.478627 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.478590 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:05.555821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555785 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555835 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555869 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555885 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555925 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gln69\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556021 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.555943 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556089 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556139 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556205 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556278 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556336 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556398 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556654 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556636 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.556745 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.556725 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.558727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.558706 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.558962 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.558943 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.619863 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.619790 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.620050 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.620032 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gln69\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\") pod \"e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:05.635196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.635139 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:05.679186 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:05.679154 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:06.107124 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:06.107100 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:18:06.108853 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:18:06.108825 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e09c3a7_31bb_4154_8a45_6f69e49c8322.slice/crio-96aecb9e2d594f1462668694d74e598349c5cf193acd67b9e965a1ff0925afb9 WatchSource:0}: Error finding container 96aecb9e2d594f1462668694d74e598349c5cf193acd67b9e965a1ff0925afb9: Status 404 returned error can't find the container with id 96aecb9e2d594f1462668694d74e598349c5cf193acd67b9e965a1ff0925afb9 Nov 03 00:18:06.111089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:06.111060 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:06.111151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:06.111128 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:06.111216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:06.111157 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:06.169156 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:06.169132 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:07.120997 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:07.120957 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerStarted","Data":"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef"} Nov 03 00:18:07.121465 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:07.121012 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerStarted","Data":"96aecb9e2d594f1462668694d74e598349c5cf193acd67b9e965a1ff0925afb9"} Nov 03 00:18:07.243929 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:07.243871 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podStartSLOduration=2.243857342 podStartE2EDuration="2.243857342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:07.241783788 +0000 UTC m=+5401.671653662" watchObservedRunningTime="2026-11-03 00:18:07.243857342 +0000 UTC m=+5401.673727256" Nov 03 00:18:07.679519 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:07.679483 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:07.684568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:07.684544 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:08.125918 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:08.125828 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:08.126928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:08.126911 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:09.259706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:09.259666 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:09.431780 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:09.431739 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:09.579103 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:09.579016 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:12.970730 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:12.970678 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:13.142235 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.142195 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:13.384026 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.383946 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:18:13.388902 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.388884 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.443165 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.443139 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-018f8381\"" Nov 03 00:18:13.493955 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.493924 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-018f8381-dockercfg-k7b42\"" Nov 03 00:18:13.495904 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.495877 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:18:13.531632 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.531598 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.531632 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.531635 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j75\" (UniqueName: \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.546827 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.546799 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:18:13.550697 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.550678 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.588984 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.588961 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:18:13.632409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.632380 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.632409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.632412 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j75\" (UniqueName: \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.632637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.632472 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh2h\" (UniqueName: \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.632637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.632501 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.633141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.633118 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.694578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.694552 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j75\" (UniqueName: \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\") pod \"payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.701349 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.701323 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:13.733310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.733272 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh2h\" (UniqueName: \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.733524 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.733321 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.734024 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.734000 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.795194 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.795134 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh2h\" (UniqueName: \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\") pod \"payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:13.863111 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:13.863076 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:14.074564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:14.074533 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:18:14.075897 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:18:14.075873 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136b55ed_1f9a_49fd_aa74_049407abd623.slice/crio-62ab4a2cdfed31fb67ec6fe6b4bfb272fd0b0eb6e01f683a51e1e58bab1e5d24 WatchSource:0}: Error finding container 62ab4a2cdfed31fb67ec6fe6b4bfb272fd0b0eb6e01f683a51e1e58bab1e5d24: Status 404 returned error can't find the container with id 62ab4a2cdfed31fb67ec6fe6b4bfb272fd0b0eb6e01f683a51e1e58bab1e5d24 Nov 03 00:18:14.159050 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:14.159018 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" event={"ID":"136b55ed-1f9a-49fd-aa74-049407abd623","Type":"ContainerStarted","Data":"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372"} Nov 03 00:18:14.159224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:14.159080 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" event={"ID":"136b55ed-1f9a-49fd-aa74-049407abd623","Type":"ContainerStarted","Data":"62ab4a2cdfed31fb67ec6fe6b4bfb272fd0b0eb6e01f683a51e1e58bab1e5d24"} Nov 03 00:18:14.159224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:14.159109 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:14.236495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:14.236470 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:18:14.238527 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:18:14.238502 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5718dd_77e7_419c_8793_4a6c79db07d5.slice/crio-f45d3c79a4f0c4a15468ebbf5223236ebcfd9c28f3b4d10fd56875f148e63332 WatchSource:0}: Error finding container f45d3c79a4f0c4a15468ebbf5223236ebcfd9c28f3b4d10fd56875f148e63332: Status 404 returned error can't find the container with id f45d3c79a4f0c4a15468ebbf5223236ebcfd9c28f3b4d10fd56875f148e63332 Nov 03 00:18:15.165262 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.165220 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" event={"ID":"ca5718dd-77e7-419c-8793-4a6c79db07d5","Type":"ContainerStarted","Data":"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9"} Nov 03 00:18:15.165711 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.165324 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" event={"ID":"ca5718dd-77e7-419c-8793-4a6c79db07d5","Type":"ContainerStarted","Data":"f45d3c79a4f0c4a15468ebbf5223236ebcfd9c28f3b4d10fd56875f148e63332"} Nov 03 00:18:15.165711 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.165505 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:15.282542 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.282489 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" podStartSLOduration=2.282475621 podStartE2EDuration="2.282475621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:14.277749308 +0000 UTC m=+5408.707619169" watchObservedRunningTime="2026-11-03 00:18:15.282475621 +0000 UTC m=+5409.712345489" Nov 03 00:18:15.283428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.283397 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" podStartSLOduration=2.283385731 podStartE2EDuration="2.283385731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:15.282263875 +0000 UTC m=+5409.712133739" watchObservedRunningTime="2026-11-03 00:18:15.283385731 +0000 UTC m=+5409.713255593" Nov 03 00:18:15.788897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.788869 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:15.948800 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:15.948759 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:16.093811 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:16.093734 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:17.558267 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:18:17.558235 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:18:20.840585 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:20.840545 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:23.818422 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:23.818384 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:18:25.168725 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:25.168698 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:18:25.882877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:25.882840 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:26.174409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:26.174384 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:18:26.456019 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:26.455930 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:30.854095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:30.854054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:33.282350 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.282272 4494 generic.go:356] "Generic (PLEG): container finished" podID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerID="d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef" exitCode=0 Nov 03 00:18:33.282779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.282345 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerDied","Data":"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef"} Nov 03 00:18:33.282779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.282723 4494 scope.go:122] "RemoveContainer" containerID="d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef" Nov 03 00:18:33.284087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.284056 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:33.284216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.284128 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:33.284216 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:33.284160 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:18:34.289655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:34.289623 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerStarted","Data":"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb"} Nov 03 00:18:34.680089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:34.680047 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:34.681248 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:34.681225 4494 patch_prober.go:29] interesting pod/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 03 00:18:34.681309 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:34.681284 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 03 00:18:35.680043 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:35.680006 4494 patch_prober.go:29] interesting pod/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 03 00:18:35.680518 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:35.680064 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 03 00:18:36.252120 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:36.252080 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:36.680487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:36.680461 4494 patch_prober.go:29] interesting pod/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 03 00:18:36.680890 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:36.680515 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 03 00:18:37.679869 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:37.679826 4494 patch_prober.go:29] interesting pod/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 03 00:18:37.680056 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:37.679888 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 03 00:18:38.126516 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:38.126416 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:38.679760 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:38.679719 4494 patch_prober.go:29] interesting pod/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body= Nov 03 00:18:38.679957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:38.679789 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" Nov 03 00:18:39.683779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:39.683749 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:39.684878 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:39.684855 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:18:41.568582 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:41.568548 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:45.841483 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.841446 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7"] Nov 03 00:18:45.843970 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.843953 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:45.899987 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.899956 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:18:45.950649 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.950614 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:18:45.952639 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.952616 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7"] Nov 03 00:18:45.969499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.969466 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8pg\" (UniqueName: \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:45.969643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.969623 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:45.969706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:45.969689 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.070603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.070570 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.070843 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.070626 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.070843 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.070670 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8pg\" (UniqueName: \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.071244 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.071221 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.073134 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.073114 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.132747 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.132670 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8pg\" (UniqueName: \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\") pod \"maas-api-revoke-keys-e2e-ait-018f8381-72bt7\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.155786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.155760 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:46.438442 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.438407 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:46.527139 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.527114 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7"] Nov 03 00:18:46.529037 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:18:46.529008 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda31599_159a_4412_980d_f8b6de3669bf.slice/crio-e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e WatchSource:0}: Error finding container e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e: Status 404 returned error can't find the container with id e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e Nov 03 00:18:46.580717 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:46.580681 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:47.354247 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:47.354218 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" event={"ID":"eda31599-159a-4412-980d-f8b6de3669bf","Type":"ContainerStarted","Data":"564d5ab783661453ffd572c5dc02a06b2f898f28fbcee43914b48c23c9dce3ce"} Nov 03 00:18:47.354648 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:47.354266 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" event={"ID":"eda31599-159a-4412-980d-f8b6de3669bf","Type":"ContainerStarted","Data":"e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e"} Nov 03 00:18:47.472568 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:47.472519 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" podStartSLOduration=2.47250365 podStartE2EDuration="2.47250365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:47.471449153 +0000 UTC m=+5441.901319016" watchObservedRunningTime="2026-11-03 00:18:47.47250365 +0000 UTC m=+5441.902373511" Nov 03 00:18:47.558106 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:18:47.558030 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:18:48.361363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:48.361328 4494 generic.go:356] "Generic (PLEG): container finished" podID="eda31599-159a-4412-980d-f8b6de3669bf" containerID="564d5ab783661453ffd572c5dc02a06b2f898f28fbcee43914b48c23c9dce3ce" exitCode=0 Nov 03 00:18:48.361768 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:48.361414 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" event={"ID":"eda31599-159a-4412-980d-f8b6de3669bf","Type":"ContainerDied","Data":"564d5ab783661453ffd572c5dc02a06b2f898f28fbcee43914b48c23c9dce3ce"} Nov 03 00:18:49.505659 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.505627 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:49.606262 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.606234 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\" (UniqueName: \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\") pod \"eda31599-159a-4412-980d-f8b6de3669bf\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " Nov 03 00:18:49.606450 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.606364 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\") pod \"eda31599-159a-4412-980d-f8b6de3669bf\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " Nov 03 00:18:49.606450 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.606399 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\") pod \"eda31599-159a-4412-980d-f8b6de3669bf\" (UID: \"eda31599-159a-4412-980d-f8b6de3669bf\") " Nov 03 00:18:49.606808 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.606777 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca" pod "eda31599-159a-4412-980d-f8b6de3669bf" (UID: "eda31599-159a-4412-980d-f8b6de3669bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:49.608599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.608574 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca" pod "eda31599-159a-4412-980d-f8b6de3669bf" (UID: "eda31599-159a-4412-980d-f8b6de3669bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:49.608710 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.608621 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg" pod "eda31599-159a-4412-980d-f8b6de3669bf" (UID: "eda31599-159a-4412-980d-f8b6de3669bf"). InnerVolumeSpecName "kube-api-access-bw8pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:49.707361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.707336 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8pg\" (UniqueName: \"kubernetes.io/projected/eda31599-159a-4412-980d-f8b6de3669bf-kube-api-access-bw8pg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:18:49.707361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.707362 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:18:49.707536 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:49.707373 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda31599-159a-4412-980d-f8b6de3669bf-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:18:50.372888 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:50.372857 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" Nov 03 00:18:50.373087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:50.372859 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7" event={"ID":"eda31599-159a-4412-980d-f8b6de3669bf","Type":"ContainerDied","Data":"e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e"} Nov 03 00:18:50.373087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:50.372978 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a7f1a4dbb53f9b88daab54a6dcac550d36796323cf258da71d54fccbf42e6e" Nov 03 00:18:51.460392 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:51.460356 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:56.456542 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:56.456505 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:57.333512 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:57.333479 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:18:57.439354 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:57.439323 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:18:57.688062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:57.688028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:18:57.833586 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:18:57.833550 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:01.241685 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:01.241648 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:02.334093 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.334022 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" podUID="ca5718dd-77e7-419c-8793-4a6c79db07d5" containerName="payload-processing" containerID="cri-o://27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9" gracePeriod=25 Nov 03 00:19:02.440331 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.440275 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" podUID="136b55ed-1f9a-49fd-aa74-049407abd623" containerName="payload-pre-processing" containerID="cri-o://ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372" gracePeriod=25 Nov 03 00:19:02.582127 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.582098 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:19:02.691630 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.691606 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:19:02.741341 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.741306 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\" (UniqueName: \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\") pod \"ca5718dd-77e7-419c-8793-4a6c79db07d5\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " Nov 03 00:19:02.741540 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.741479 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\") pod \"ca5718dd-77e7-419c-8793-4a6c79db07d5\" (UID: \"ca5718dd-77e7-419c-8793-4a6c79db07d5\") " Nov 03 00:19:02.741838 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.741809 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume" pod "ca5718dd-77e7-419c-8793-4a6c79db07d5" (UID: "ca5718dd-77e7-419c-8793-4a6c79db07d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:02.743537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.743519 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h" pod "ca5718dd-77e7-419c-8793-4a6c79db07d5" (UID: "ca5718dd-77e7-419c-8793-4a6c79db07d5"). InnerVolumeSpecName "kube-api-access-7dh2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:02.842312 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.842281 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\") pod \"136b55ed-1f9a-49fd-aa74-049407abd623\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " Nov 03 00:19:02.842490 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.842338 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\" (UniqueName: \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\") pod \"136b55ed-1f9a-49fd-aa74-049407abd623\" (UID: \"136b55ed-1f9a-49fd-aa74-049407abd623\") " Nov 03 00:19:02.842610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.842590 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5718dd-77e7-419c-8793-4a6c79db07d5-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.842651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.842616 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh2h\" (UniqueName: \"kubernetes.io/projected/ca5718dd-77e7-419c-8793-4a6c79db07d5-kube-api-access-7dh2h\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.842763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.842725 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume" pod "136b55ed-1f9a-49fd-aa74-049407abd623" (UID: "136b55ed-1f9a-49fd-aa74-049407abd623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:02.844680 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.844660 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75" pod "136b55ed-1f9a-49fd-aa74-049407abd623" (UID: "136b55ed-1f9a-49fd-aa74-049407abd623"). InnerVolumeSpecName "kube-api-access-p4j75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:02.943700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.943669 4494 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136b55ed-1f9a-49fd-aa74-049407abd623-plugins-config-volume\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:02.943700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:02.943694 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4j75\" (UniqueName: \"kubernetes.io/projected/136b55ed-1f9a-49fd-aa74-049407abd623-kube-api-access-p4j75\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:03.446536 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.446504 4494 generic.go:356] "Generic (PLEG): container finished" podID="ca5718dd-77e7-419c-8793-4a6c79db07d5" containerID="27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9" exitCode=0 Nov 03 00:19:03.447057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.446563 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" Nov 03 00:19:03.447057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.446601 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" event={"ID":"ca5718dd-77e7-419c-8793-4a6c79db07d5","Type":"ContainerDied","Data":"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9"} Nov 03 00:19:03.447057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.446653 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs" event={"ID":"ca5718dd-77e7-419c-8793-4a6c79db07d5","Type":"ContainerDied","Data":"f45d3c79a4f0c4a15468ebbf5223236ebcfd9c28f3b4d10fd56875f148e63332"} Nov 03 00:19:03.447057 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.446682 4494 scope.go:122] "RemoveContainer" containerID="27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9" Nov 03 00:19:03.448069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.448044 4494 generic.go:356] "Generic (PLEG): container finished" podID="136b55ed-1f9a-49fd-aa74-049407abd623" containerID="ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372" exitCode=0 Nov 03 00:19:03.448148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.448085 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" Nov 03 00:19:03.448148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.448119 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" event={"ID":"136b55ed-1f9a-49fd-aa74-049407abd623","Type":"ContainerDied","Data":"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372"} Nov 03 00:19:03.448343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.448154 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn" event={"ID":"136b55ed-1f9a-49fd-aa74-049407abd623","Type":"ContainerDied","Data":"62ab4a2cdfed31fb67ec6fe6b4bfb272fd0b0eb6e01f683a51e1e58bab1e5d24"} Nov 03 00:19:03.457390 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.457375 4494 scope.go:122] "RemoveContainer" containerID="27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9" Nov 03 00:19:03.457693 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:03.457672 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9\": container with ID starting with 27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9 not found: ID does not exist" containerID="27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9" Nov 03 00:19:03.457792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.457700 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9"} err="failed to get container status \"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9\": rpc error: code = NotFound desc = could not find container \"27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9\": container with ID starting with 27af4a4b2fcfd5f9b8b4680120a77b63193553ee02b1bd29ea023340207684c9 not found: ID does not exist" Nov 03 00:19:03.457792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.457717 4494 scope.go:122] "RemoveContainer" containerID="ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372" Nov 03 00:19:03.465577 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.465560 4494 scope.go:122] "RemoveContainer" containerID="ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372" Nov 03 00:19:03.465831 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:03.465813 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372\": container with ID starting with ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372 not found: ID does not exist" containerID="ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372" Nov 03 00:19:03.465887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.465838 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372"} err="failed to get container status \"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372\": rpc error: code = NotFound desc = could not find container \"ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372\": container with ID starting with ad01230538ea487c4e502a08762f96118a34b59f6faa4b9d530882372157b372 not found: ID does not exist" Nov 03 00:19:03.613005 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.612926 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:19:03.619406 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.619379 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-018f8381-75f6dbb459-gb6zn"] Nov 03 00:19:03.692612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.692579 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136b55ed-1f9a-49fd-aa74-049407abd623" path="/var/lib/kubelet/pods/136b55ed-1f9a-49fd-aa74-049407abd623/volumes" Nov 03 00:19:03.729356 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.729322 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:19:03.733678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:03.733653 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-018f8381-5d76967b5c-dshqs"] Nov 03 00:19:05.570468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:05.570440 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:05.693516 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:05.693480 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5718dd-77e7-419c-8793-4a6c79db07d5" path="/var/lib/kubelet/pods/ca5718dd-77e7-419c-8793-4a6c79db07d5/volumes" Nov 03 00:19:07.110850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:07.110817 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:07.391121 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:07.391038 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:07.603651 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:07.603574 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7"] Nov 03 00:19:07.654334 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:07.654263 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-018f8381-72bt7"] Nov 03 00:19:07.691499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:07.691470 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda31599-159a-4412-980d-f8b6de3669bf" path="/var/lib/kubelet/pods/eda31599-159a-4412-980d-f8b6de3669bf/volumes" Nov 03 00:19:10.070975 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:10.070944 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:19:12.072152 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:12.072119 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:19:12.072621 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:12.072334 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" containerID="cri-o://cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb" gracePeriod=30 Nov 03 00:19:12.076960 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:12.076933 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:16.438053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.438017 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 03 00:19:16.438544 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.438239 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7457749d7d-nzlvl" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" containerID="cri-o://e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a" gracePeriod=30 Nov 03 00:19:16.685041 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.685016 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 03 00:19:16.764022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.763948 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\") pod \"d98647bf-90df-4e01-8959-b857f423b272\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " Nov 03 00:19:16.764202 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.764048 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\") pod \"d98647bf-90df-4e01-8959-b857f423b272\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " Nov 03 00:19:16.764202 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.764083 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\" (UniqueName: \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\") pod \"d98647bf-90df-4e01-8959-b857f423b272\" (UID: \"d98647bf-90df-4e01-8959-b857f423b272\") " Nov 03 00:19:16.766265 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.766239 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert" pod "d98647bf-90df-4e01-8959-b857f423b272" (UID: "d98647bf-90df-4e01-8959-b857f423b272"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:16.766743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.766725 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n" pod "d98647bf-90df-4e01-8959-b857f423b272" (UID: "d98647bf-90df-4e01-8959-b857f423b272"). InnerVolumeSpecName "kube-api-access-rll2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:16.766743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.766731 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs" pod "d98647bf-90df-4e01-8959-b857f423b272" (UID: "d98647bf-90df-4e01-8959-b857f423b272"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:16.865727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.865696 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:16.865727 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.865725 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rll2n\" (UniqueName: \"kubernetes.io/projected/d98647bf-90df-4e01-8959-b857f423b272-kube-api-access-rll2n\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:16.865956 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:16.865736 4494 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d98647bf-90df-4e01-8959-b857f423b272-webhook-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.303655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.303631 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:19:17.369790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369720 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.369790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369756 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.369790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369784 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369817 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369844 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369876 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.369951 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370027 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370027 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:17.370100 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370041 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:17.370485 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370155 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\") pod \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\" (UID: \"2e09c3a7-31bb-4154-8a45-6f69e49c8322\") " Nov 03 00:19:17.370485 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370188 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:17.370485 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370244 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:17.370485 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370372 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:17.370713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370512 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.370713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370531 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.370713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370545 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.370713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370558 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.370713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.370571 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.372210 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.372161 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:19:17.372344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.372245 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:17.372344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.372250 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:17.372344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.372292 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69" pod "2e09c3a7-31bb-4154-8a45-6f69e49c8322" (UID: "2e09c3a7-31bb-4154-8a45-6f69e49c8322"). InnerVolumeSpecName "kube-api-access-gln69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:17.471343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.471306 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.471343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.471334 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.471343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.471344 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gln69\" (UniqueName: \"kubernetes.io/projected/2e09c3a7-31bb-4154-8a45-6f69e49c8322-kube-api-access-gln69\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.471343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.471353 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e09c3a7-31bb-4154-8a45-6f69e49c8322-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:17.523387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.523351 4494 generic.go:356] "Generic (PLEG): container finished" podID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerID="cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb" exitCode=0 Nov 03 00:19:17.523584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.523399 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" Nov 03 00:19:17.523584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.523437 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerDied","Data":"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb"} Nov 03 00:19:17.523584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.523503 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v" event={"ID":"2e09c3a7-31bb-4154-8a45-6f69e49c8322","Type":"ContainerDied","Data":"96aecb9e2d594f1462668694d74e598349c5cf193acd67b9e965a1ff0925afb9"} Nov 03 00:19:17.523584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.523538 4494 scope.go:122] "RemoveContainer" containerID="cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb" Nov 03 00:19:17.525310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.525286 4494 generic.go:356] "Generic (PLEG): container finished" podID="d98647bf-90df-4e01-8959-b857f423b272" containerID="e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a" exitCode=0 Nov 03 00:19:17.525466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.525335 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-nzlvl" Nov 03 00:19:17.525466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.525374 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerDied","Data":"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a"} Nov 03 00:19:17.525466 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.525440 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-nzlvl" event={"ID":"d98647bf-90df-4e01-8959-b857f423b272","Type":"ContainerDied","Data":"4cbf02ffb159e0deccf1e373ea4f5ca576e7aeb135cd574541e1a3b9c2eea00a"} Nov 03 00:19:17.534473 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.534449 4494 scope.go:122] "RemoveContainer" containerID="d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef" Nov 03 00:19:17.560529 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.560504 4494 scope.go:122] "RemoveContainer" containerID="cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb" Nov 03 00:19:17.560890 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:17.560868 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb\": container with ID starting with cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb not found: ID does not exist" containerID="cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb" Nov 03 00:19:17.560984 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.560900 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb"} err="failed to get container status \"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb\": rpc error: code = NotFound desc = could not find container \"cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb\": container with ID starting with cc05f34efd3528d3b44036a73cce24e7e329b511516896c9a3ecd58ecb290dbb not found: ID does not exist" Nov 03 00:19:17.560984 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.560918 4494 scope.go:122] "RemoveContainer" containerID="d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef" Nov 03 00:19:17.561246 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:17.561229 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef\": container with ID starting with d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef not found: ID does not exist" containerID="d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef" Nov 03 00:19:17.561319 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.561251 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef"} err="failed to get container status \"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef\": rpc error: code = NotFound desc = could not find container \"d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef\": container with ID starting with d9cddaa258e808c0459812fa49b6125433d741ac57adc884a5975a3be4d838ef not found: ID does not exist" Nov 03 00:19:17.561319 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.561273 4494 scope.go:122] "RemoveContainer" containerID="e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a" Nov 03 00:19:17.561428 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:17.561352 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:19:17.572589 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.572564 4494 scope.go:122] "RemoveContainer" containerID="48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b" Nov 03 00:19:17.581062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.581042 4494 scope.go:122] "RemoveContainer" containerID="e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a" Nov 03 00:19:17.581390 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:17.581365 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a\": container with ID starting with e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a not found: ID does not exist" containerID="e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a" Nov 03 00:19:17.581487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.581404 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a"} err="failed to get container status \"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a\": rpc error: code = NotFound desc = could not find container \"e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a\": container with ID starting with e62e9644623e67519c3e1529ff252ed4f342bd5f731d2fa6b934db97434d9d7a not found: ID does not exist" Nov 03 00:19:17.581487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.581427 4494 scope.go:122] "RemoveContainer" containerID="48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b" Nov 03 00:19:17.581715 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:17.581697 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b\": container with ID starting with 48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b not found: ID does not exist" containerID="48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b" Nov 03 00:19:17.581781 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.581722 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b"} err="failed to get container status \"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b\": rpc error: code = NotFound desc = could not find container \"48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b\": container with ID starting with 48ed06c0e9642cbe6b619224a35ca00080d3929c0b0b62138d3f6bafc86b9c1b not found: ID does not exist" Nov 03 00:19:17.693643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.693610 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:19:17.695825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.695801 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-018f8381-openshift-default-7c97f8c955-twb6v"] Nov 03 00:19:17.805514 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.805483 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 03 00:19:17.809304 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:17.809278 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7457749d7d-nzlvl"] Nov 03 00:19:18.664138 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664097 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:18.664694 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664675 4494 cpu_manager.go:414] "containerMap: removing container" podUID="136b55ed-1f9a-49fd-aa74-049407abd623" containerName="payload-pre-processing" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664697 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136b55ed-1f9a-49fd-aa74-049407abd623" containerName="payload-pre-processing" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664714 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664721 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664755 4494 cpu_manager.go:414] "containerMap: removing container" podUID="ca5718dd-77e7-419c-8793-4a6c79db07d5" containerName="payload-processing" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664763 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5718dd-77e7-419c-8793-4a6c79db07d5" containerName="payload-processing" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664772 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664779 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664804 4494 cpu_manager.go:414] "containerMap: removing container" podUID="eda31599-159a-4412-980d-f8b6de3669bf" containerName="revoke-keys" Nov 03 00:19:18.664816 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664812 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda31599-159a-4412-980d-f8b6de3669bf" containerName="revoke-keys" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664911 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664928 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5718dd-77e7-419c-8793-4a6c79db07d5" containerName="payload-processing" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664937 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664952 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664961 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda31599-159a-4412-980d-f8b6de3669bf" containerName="revoke-keys" Nov 03 00:19:18.665308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.664973 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="136b55ed-1f9a-49fd-aa74-049407abd623" containerName="payload-pre-processing" Nov 03 00:19:18.669750 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.669729 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.773755 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773719 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:19:18.773961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773800 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:19:18.773961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773808 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:19:18.773961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773840 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:19:18.773961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773847 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:19:18.773961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.773884 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p68w6\"" Nov 03 00:19:18.775974 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.775952 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:18.784528 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.784509 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.784592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.784537 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.784592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.784558 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxqx\" (UniqueName: \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.885882 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.885848 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.886081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.885888 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.886081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.885926 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxqx\" (UniqueName: \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.888489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.888463 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.888611 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.888569 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.948367 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.948297 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxqx\" (UniqueName: \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\") pod \"maas-controller-7457749d7d-hkrxd\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:18.981853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:18.981819 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:19.355744 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:19.355721 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:19.357478 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:19:19.357444 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4f3c14_fb3b_425e_b733_8b745ab47ecc.slice/crio-26c6fa8c3d0eb2f2c33c23c1672328362a85fb51350064e8ceaa749ee4c20610 WatchSource:0}: Error finding container 26c6fa8c3d0eb2f2c33c23c1672328362a85fb51350064e8ceaa749ee4c20610: Status 404 returned error can't find the container with id 26c6fa8c3d0eb2f2c33c23c1672328362a85fb51350064e8ceaa749ee4c20610 Nov 03 00:19:19.358668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:19.358652 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:19:19.539391 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:19.539354 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-hkrxd" event={"ID":"5a4f3c14-fb3b-425e-b733-8b745ab47ecc","Type":"ContainerStarted","Data":"26c6fa8c3d0eb2f2c33c23c1672328362a85fb51350064e8ceaa749ee4c20610"} Nov 03 00:19:19.692047 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:19.692005 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" path="/var/lib/kubelet/pods/2e09c3a7-31bb-4154-8a45-6f69e49c8322/volumes" Nov 03 00:19:19.692615 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:19.692598 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98647bf-90df-4e01-8959-b857f423b272" path="/var/lib/kubelet/pods/d98647bf-90df-4e01-8959-b857f423b272/volumes" Nov 03 00:19:20.545986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:20.545894 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-hkrxd" event={"ID":"5a4f3c14-fb3b-425e-b733-8b745ab47ecc","Type":"ContainerStarted","Data":"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f"} Nov 03 00:19:20.545986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:20.545965 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:20.665217 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:20.665146 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7457749d7d-hkrxd" podStartSLOduration=1.8246698970000002 podStartE2EDuration="2.665127978s" totalImagesPullingTime="840.458081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:20.664397031 +0000 UTC m=+5475.094266892" watchObservedRunningTime="2026-11-03 00:19:20.665127978 +0000 UTC m=+5475.094997843" Nov 03 00:19:26.784009 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:26.783973 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:31.553213 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:31.553160 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:39.187448 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.187411 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw"] Nov 03 00:19:39.188039 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.188018 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:39.188135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.188042 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e09c3a7-31bb-4154-8a45-6f69e49c8322" containerName="istio-proxy" Nov 03 00:19:39.188135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.188094 4494 cpu_manager.go:414] "containerMap: removing container" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:39.188135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.188103 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:39.188321 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.188237 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98647bf-90df-4e01-8959-b857f423b272" containerName="manager" Nov 03 00:19:39.191632 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.191613 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.290179 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.290143 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.290386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.290209 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvfr\" (UniqueName: \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.290451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.290377 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.296425 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.296402 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:19:39.296551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.296460 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:39.298343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.298316 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw"] Nov 03 00:19:39.391823 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.391795 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.392039 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.391832 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvfr\" (UniqueName: \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.392039 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.391913 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.392565 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.392543 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.394355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.394332 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.453806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.453734 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvfr\" (UniqueName: \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\") pod \"maas-api-revoke-sub-85237e45-2fmkw\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.504462 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.504418 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:39.877985 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:39.877957 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw"] Nov 03 00:19:39.879589 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:19:39.879556 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d736409_4c28_4871_927a_8c6d4e75c808.slice/crio-c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840 WatchSource:0}: Error finding container c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840: Status 404 returned error can't find the container with id c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840 Nov 03 00:19:40.586548 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.586516 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss"] Nov 03 00:19:40.590088 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.590061 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.602613 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.602592 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh"] Nov 03 00:19:40.605875 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.605860 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.643238 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.643201 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-98x6w\"" Nov 03 00:19:40.645422 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.645358 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" event={"ID":"7d736409-4c28-4871-927a-8c6d4e75c808","Type":"ContainerStarted","Data":"254e969671221b13b222e5b2c0e8ce7dfd3f8babe274ceff6bd7005945e5dcc0"} Nov 03 00:19:40.645592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.645450 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" event={"ID":"7d736409-4c28-4871-927a-8c6d4e75c808","Type":"ContainerStarted","Data":"c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840"} Nov 03 00:19:40.689154 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.689122 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss"] Nov 03 00:19:40.704187 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704137 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704342 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704202 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704342 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704298 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704342 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704335 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704459 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704366 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704470 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704495 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704572 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704562 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704590 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704644 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704612 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704644 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704633 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wbm\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-kube-api-access-z7wbm\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704709 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704676 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704745 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704707 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfv7\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-kube-api-access-lpfv7\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704756 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b38964-2c95-47b9-bea3-6a2d26f224cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704830 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704789 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704830 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704810 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.704897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704827 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.704935 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.704913 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.739391 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.739356 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh"] Nov 03 00:19:40.806069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806025 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806298 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806074 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806298 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806221 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806298 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806267 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wbm\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-kube-api-access-z7wbm\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806319 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806352 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfv7\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-kube-api-access-lpfv7\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806389 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b38964-2c95-47b9-bea3-6a2d26f224cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806416 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806445 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806489 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806517 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806524 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806538 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806675 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806702 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806735 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806782 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.806826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806811 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.807315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806848 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.807315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.806872 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.807315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.807239 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b38964-2c95-47b9-bea3-6a2d26f224cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.807484 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.807414 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.807556 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.807485 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810205 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810326 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810353 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b38964-2c95-47b9-bea3-6a2d26f224cc-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810427 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810553 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810603 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810653 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810718 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.821214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.810974 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.871660 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.871625 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.871815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.871719 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfv7\" (UniqueName: \"kubernetes.io/projected/bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c-kube-api-access-lpfv7\") pod \"maas-default-gateway-openshift-default-694db55c87-6wkzh\" (UID: \"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.871815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.871760 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wbm\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-kube-api-access-z7wbm\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.872281 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.872263 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b38964-2c95-47b9-bea3-6a2d26f224cc-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-k2dss\" (UID: \"80b38964-2c95-47b9-bea3-6a2d26f224cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.903825 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.903798 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:40.917772 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.917748 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:40.944212 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:40.943564 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" podStartSLOduration=1.9435438980000002 podStartE2EDuration="1.943543898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:40.943328133 +0000 UTC m=+5495.373198033" watchObservedRunningTime="2026-11-03 00:19:40.943543898 +0000 UTC m=+5495.373413760" Nov 03 00:19:41.331189 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.331149 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss"] Nov 03 00:19:41.333520 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:19:41.332581 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b38964_2c95_47b9_bea3_6a2d26f224cc.slice/crio-db641c811d166ad913329ccf308df357d6b6a2b33fa79858050818d4abb9a3a6 WatchSource:0}: Error finding container db641c811d166ad913329ccf308df357d6b6a2b33fa79858050818d4abb9a3a6: Status 404 returned error can't find the container with id db641c811d166ad913329ccf308df357d6b6a2b33fa79858050818d4abb9a3a6 Nov 03 00:19:41.333786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.333762 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh"] Nov 03 00:19:41.335000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.334970 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.335091 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.335037 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.335091 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:19:41.335045 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf94d8f2_7b5b_4f04_88d7_2639a4ee6b4c.slice/crio-5cec711d32cc5206bdbbb1189dcf2b7b96c1ba05ea76d779fa32c5f9a80d9c47 WatchSource:0}: Error finding container 5cec711d32cc5206bdbbb1189dcf2b7b96c1ba05ea76d779fa32c5f9a80d9c47: Status 404 returned error can't find the container with id 5cec711d32cc5206bdbbb1189dcf2b7b96c1ba05ea76d779fa32c5f9a80d9c47 Nov 03 00:19:41.335091 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.335070 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.337319 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.337290 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.337390 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.337371 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.337433 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.337422 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:41.652909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.652808 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" event={"ID":"80b38964-2c95-47b9-bea3-6a2d26f224cc","Type":"ContainerStarted","Data":"bef8bab70a1cd14fc39e8771cc1d30caffd5fe1c5ba42380e8050e1e1a478043"} Nov 03 00:19:41.652909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.652876 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" event={"ID":"80b38964-2c95-47b9-bea3-6a2d26f224cc","Type":"ContainerStarted","Data":"db641c811d166ad913329ccf308df357d6b6a2b33fa79858050818d4abb9a3a6"} Nov 03 00:19:41.655366 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.655330 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" event={"ID":"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c","Type":"ContainerStarted","Data":"7cec2ce44c398f1d7f474d918f14fecf548f1b345606cc83b14afaa8a0cb80e2"} Nov 03 00:19:41.655557 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.655395 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" event={"ID":"bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c","Type":"ContainerStarted","Data":"5cec711d32cc5206bdbbb1189dcf2b7b96c1ba05ea76d779fa32c5f9a80d9c47"} Nov 03 00:19:41.657437 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.657412 4494 generic.go:356] "Generic (PLEG): container finished" podID="7d736409-4c28-4871-927a-8c6d4e75c808" containerID="254e969671221b13b222e5b2c0e8ce7dfd3f8babe274ceff6bd7005945e5dcc0" exitCode=0 Nov 03 00:19:41.657539 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.657448 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" event={"ID":"7d736409-4c28-4871-927a-8c6d4e75c808","Type":"ContainerDied","Data":"254e969671221b13b222e5b2c0e8ce7dfd3f8babe274ceff6bd7005945e5dcc0"} Nov 03 00:19:41.774712 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.774644 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podStartSLOduration=1.7746291730000001 podStartE2EDuration="1.774629173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:41.773628229 +0000 UTC m=+5496.203498091" watchObservedRunningTime="2026-11-03 00:19:41.774629173 +0000 UTC m=+5496.204499033" Nov 03 00:19:41.904470 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.904393 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:41.918455 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:41.918417 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:42.015913 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.015857 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podStartSLOduration=2.015842152 podStartE2EDuration="2.015842152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:42.01497207 +0000 UTC m=+5496.444841946" watchObservedRunningTime="2026-11-03 00:19:42.015842152 +0000 UTC m=+5496.445712013" Nov 03 00:19:42.891846 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.891819 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:42.906007 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.905981 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:42.906148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.906030 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:42.919095 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.919073 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:42.919159 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:42.919107 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:43.043273 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.031513 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\") pod \"7d736409-4c28-4871-927a-8c6d4e75c808\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " Nov 03 00:19:43.043273 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.031688 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\") pod \"7d736409-4c28-4871-927a-8c6d4e75c808\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " Nov 03 00:19:43.043273 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.031732 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\" (UniqueName: \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\") pod \"7d736409-4c28-4871-927a-8c6d4e75c808\" (UID: \"7d736409-4c28-4871-927a-8c6d4e75c808\") " Nov 03 00:19:43.043273 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.038354 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca" pod "7d736409-4c28-4871-927a-8c6d4e75c808" (UID: "7d736409-4c28-4871-927a-8c6d4e75c808"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:43.083772 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.076414 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr" pod "7d736409-4c28-4871-927a-8c6d4e75c808" (UID: "7d736409-4c28-4871-927a-8c6d4e75c808"). InnerVolumeSpecName "kube-api-access-xxvfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:43.086135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.086105 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca" pod "7d736409-4c28-4871-927a-8c6d4e75c808" (UID: "7d736409-4c28-4871-927a-8c6d4e75c808"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:43.154196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.142049 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.154196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.142086 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvfr\" (UniqueName: \"kubernetes.io/projected/7d736409-4c28-4871-927a-8c6d4e75c808-kube-api-access-xxvfr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.154196 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.142104 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d736409-4c28-4871-927a-8c6d4e75c808-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.689198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.688629 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" Nov 03 00:19:43.693849 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.693821 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw" event={"ID":"7d736409-4c28-4871-927a-8c6d4e75c808","Type":"ContainerDied","Data":"c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840"} Nov 03 00:19:43.694002 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.693863 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2978b13933a8f765a5bb4ba1cb7fcda4b654e9fb3869b4284c164ec851ae840" Nov 03 00:19:43.904928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.904886 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:43.905451 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.904939 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:43.918359 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.918328 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:43.918487 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:43.918377 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:44.904624 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:44.904584 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:44.904834 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:44.904647 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:44.918981 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:44.918957 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:44.919317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:44.918995 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:45.904270 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:45.904232 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:45.904471 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:45.904292 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:45.918643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:45.918606 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:45.918789 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:45.918667 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:46.788005 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:46.787964 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:46.904581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:46.904543 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:46.904779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:46.904596 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:46.919048 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:46.919018 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:46.919207 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:46.919068 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:47.560354 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:47.560307 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:19:47.904876 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:47.904782 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:47.904876 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:47.904856 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:47.919286 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:47.919246 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:47.919452 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:47.919318 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:48.904381 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:48.904347 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:19:48.904582 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:48.904400 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": dial tcp 10.133.1.52:15021: connect: connection refused" Nov 03 00:19:48.918856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:48.918819 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Nov 03 00:19:48.919214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:48.918892 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Nov 03 00:19:50.945420 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:50.945377 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:19:50.945992 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:50.945448 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:19:50.959382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:50.959330 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:19:50.959669 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:50.959625 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:19:51.597621 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:51.597535 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:51.711067 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:51.711036 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:51.711280 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:51.711245 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7457749d7d-hkrxd" podUID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" containerName="manager" containerID="cri-o://fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f" gracePeriod=30 Nov 03 00:19:51.987420 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:51.986943 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:19:51.987420 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:51.987008 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:19:52.006513 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.006302 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:19:52.006513 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.006359 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:19:52.009109 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.009086 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:52.009759 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.009740 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:52.020449 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.020422 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" Nov 03 00:19:52.033010 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.032972 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:52.065819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.065787 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\" (UniqueName: \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\") pod \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " Nov 03 00:19:52.066042 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.065862 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\") pod \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " Nov 03 00:19:52.066042 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.065897 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\") pod \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\" (UID: \"5a4f3c14-fb3b-425e-b733-8b745ab47ecc\") " Nov 03 00:19:52.068328 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.068292 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert" pod "5a4f3c14-fb3b-425e-b733-8b745ab47ecc" (UID: "5a4f3c14-fb3b-425e-b733-8b745ab47ecc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:52.069747 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.068693 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx" pod "5a4f3c14-fb3b-425e-b733-8b745ab47ecc" (UID: "5a4f3c14-fb3b-425e-b733-8b745ab47ecc"). InnerVolumeSpecName "kube-api-access-zdxqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:52.069747 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.069117 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs" pod "5a4f3c14-fb3b-425e-b733-8b745ab47ecc" (UID: "5a4f3c14-fb3b-425e-b733-8b745ab47ecc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:52.167700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.167619 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxqx\" (UniqueName: \"kubernetes.io/projected/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-kube-api-access-zdxqx\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:52.167700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.167647 4494 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-webhook-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:52.167700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.167656 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4f3c14-fb3b-425e-b733-8b745ab47ecc-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:52.178501 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.178476 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:52.178783 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.178762 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:52.179437 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.179421 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" Nov 03 00:19:52.746972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.746937 4494 generic.go:356] "Generic (PLEG): container finished" podID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" containerID="fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f" exitCode=0 Nov 03 00:19:52.747164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.747000 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-hkrxd" Nov 03 00:19:52.747164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.747020 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-hkrxd" event={"ID":"5a4f3c14-fb3b-425e-b733-8b745ab47ecc","Type":"ContainerDied","Data":"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f"} Nov 03 00:19:52.747164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.747079 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-hkrxd" event={"ID":"5a4f3c14-fb3b-425e-b733-8b745ab47ecc","Type":"ContainerDied","Data":"26c6fa8c3d0eb2f2c33c23c1672328362a85fb51350064e8ceaa749ee4c20610"} Nov 03 00:19:52.747164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.747107 4494 scope.go:122] "RemoveContainer" containerID="fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f" Nov 03 00:19:52.757078 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.757062 4494 scope.go:122] "RemoveContainer" containerID="fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f" Nov 03 00:19:52.757388 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:19:52.757368 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f\": container with ID starting with fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f not found: ID does not exist" containerID="fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f" Nov 03 00:19:52.757452 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.757398 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f"} err="failed to get container status \"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f\": rpc error: code = NotFound desc = could not find container \"fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f\": container with ID starting with fa0e25ce7e955644f66f59ed9360cb440e4596e6b6e8656127a05f7a6f894a9f not found: ID does not exist" Nov 03 00:19:52.914283 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.914252 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:52.917812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:52.917791 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7457749d7d-hkrxd"] Nov 03 00:19:53.692521 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:53.692485 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" path="/var/lib/kubelet/pods/5a4f3c14-fb3b-425e-b733-8b745ab47ecc/volumes" Nov 03 00:19:54.157460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.157362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:19:54.295728 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.295693 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7457749d7d-vhchr"] Nov 03 00:19:54.296106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296095 4494 cpu_manager.go:414] "containerMap: removing container" podUID="7d736409-4c28-4871-927a-8c6d4e75c808" containerName="revoke-keys" Nov 03 00:19:54.296224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296108 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d736409-4c28-4871-927a-8c6d4e75c808" containerName="revoke-keys" Nov 03 00:19:54.296224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296130 4494 cpu_manager.go:414] "containerMap: removing container" podUID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" containerName="manager" Nov 03 00:19:54.296224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296135 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" containerName="manager" Nov 03 00:19:54.296224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296207 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d736409-4c28-4871-927a-8c6d4e75c808" containerName="revoke-keys" Nov 03 00:19:54.296224 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.296219 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4f3c14-fb3b-425e-b733-8b745ab47ecc" containerName="manager" Nov 03 00:19:54.299275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.299256 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.352701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.352676 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:19:54.352701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.352677 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:19:54.388975 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.388946 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-webhook-cert\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.389126 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.388993 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-metrics-certs\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.389126 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.389095 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8ph\" (UniqueName: \"kubernetes.io/projected/ec7af7fd-654f-45db-a758-9307fb08247e-kube-api-access-gs8ph\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.403488 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.403453 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:19:54.403488 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.403464 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p68w6\"" Nov 03 00:19:54.403488 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.403487 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:19:54.403713 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.403527 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:19:54.405840 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.405820 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-vhchr"] Nov 03 00:19:54.490594 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.490563 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-webhook-cert\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.490793 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.490615 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-metrics-certs\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.490793 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.490660 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8ph\" (UniqueName: \"kubernetes.io/projected/ec7af7fd-654f-45db-a758-9307fb08247e-kube-api-access-gs8ph\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.493234 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.493203 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-webhook-cert\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.493370 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.493253 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec7af7fd-654f-45db-a758-9307fb08247e-metrics-certs\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.552735 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.552709 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8ph\" (UniqueName: \"kubernetes.io/projected/ec7af7fd-654f-45db-a758-9307fb08247e-kube-api-access-gs8ph\") pod \"maas-controller-7457749d7d-vhchr\" (UID: \"ec7af7fd-654f-45db-a758-9307fb08247e\") " pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.611721 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.611691 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:54.984431 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:54.984405 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7457749d7d-vhchr"] Nov 03 00:19:54.986348 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:19:54.986323 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7af7fd_654f_45db_a758_9307fb08247e.slice/crio-4f0c2f009d5b24182f02bbf6a8540c2faa3e3bbf73ff1557c73f877b02b8a298 WatchSource:0}: Error finding container 4f0c2f009d5b24182f02bbf6a8540c2faa3e3bbf73ff1557c73f877b02b8a298: Status 404 returned error can't find the container with id 4f0c2f009d5b24182f02bbf6a8540c2faa3e3bbf73ff1557c73f877b02b8a298 Nov 03 00:19:55.764951 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:55.764909 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-vhchr" event={"ID":"ec7af7fd-654f-45db-a758-9307fb08247e","Type":"ContainerStarted","Data":"4f0c2f009d5b24182f02bbf6a8540c2faa3e3bbf73ff1557c73f877b02b8a298"} Nov 03 00:19:56.779920 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:56.779888 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7457749d7d-vhchr" event={"ID":"ec7af7fd-654f-45db-a758-9307fb08247e","Type":"ContainerStarted","Data":"3d3956305f6046226acaa26f018f4f5bca834ae4e6467a78f631ae3992ba86d1"} Nov 03 00:19:56.780363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:56.779978 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:19:56.900575 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:19:56.900519 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7457749d7d-vhchr" podStartSLOduration=2.093098952 podStartE2EDuration="2.900502594s" totalImagesPullingTime="807.403642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:56.899665418 +0000 UTC m=+5511.329535280" watchObservedRunningTime="2026-11-03 00:19:56.900502594 +0000 UTC m=+5511.330372755" Nov 03 00:20:07.787769 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:07.787735 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7457749d7d-vhchr" Nov 03 00:20:17.558279 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:20:17.558200 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:20:20.046404 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:20.046365 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z"] Nov 03 00:20:20.096775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:20.096742 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4f3be0-clj8z"] Nov 03 00:20:20.193301 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:20.193267 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:20:21.696551 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:21.696050 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d621b40-5cf8-4260-b41b-76367c4bc501" path="/var/lib/kubelet/pods/2d621b40-5cf8-4260-b41b-76367c4bc501/volumes" Nov 03 00:20:27.131242 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:27.131208 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:39.884564 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:39.884526 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:20:41.063862 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.063834 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5"] Nov 03 00:20:41.067876 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.067851 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181230 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181267 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181370 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181408 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181430 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181449 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181556 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181586 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztdn\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-kube-api-access-4ztdn\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.181614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.181611 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.268944 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.268884 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5"] Nov 03 00:20:41.282450 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282423 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282455 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282495 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282528 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282557 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282580 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.282678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282616 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.282956 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283000 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283023 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283098 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztdn\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-kube-api-access-4ztdn\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283152 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283537 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283517 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.283628 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.283599 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.285125 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.285096 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.285367 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.285347 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.344126 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.344062 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.344474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.344456 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztdn\" (UniqueName: \"kubernetes.io/projected/ec3b4496-e1c1-464c-bbfb-d290f8815c66-kube-api-access-4ztdn\") pod \"maas-default-gateway-openshift-default-694db55c87-m8ss5\" (UID: \"ec3b4496-e1c1-464c-bbfb-d290f8815c66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.380474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.380452 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:41.804774 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.804750 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5"] Nov 03 00:20:41.805813 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:20:41.805786 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3b4496_e1c1_464c_bbfb_d290f8815c66.slice/crio-ce671bbf3c7c6aa2d3405bafa37e46ceb0536319bb5f0adc596474ea5037007c WatchSource:0}: Error finding container ce671bbf3c7c6aa2d3405bafa37e46ceb0536319bb5f0adc596474ea5037007c: Status 404 returned error can't find the container with id ce671bbf3c7c6aa2d3405bafa37e46ceb0536319bb5f0adc596474ea5037007c Nov 03 00:20:41.808082 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.808049 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:41.808211 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.808125 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:41.808211 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:41.808188 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:42.002984 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.002953 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" event={"ID":"ec3b4496-e1c1-464c-bbfb-d290f8815c66","Type":"ContainerStarted","Data":"7649cd2eebf4a98ef05b087252d67aeb2a5318a5d9a7901f92bd95b0ab37c797"} Nov 03 00:20:42.003154 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.003007 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" event={"ID":"ec3b4496-e1c1-464c-bbfb-d290f8815c66","Type":"ContainerStarted","Data":"ce671bbf3c7c6aa2d3405bafa37e46ceb0536319bb5f0adc596474ea5037007c"} Nov 03 00:20:42.126357 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.126245 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podStartSLOduration=1.126230818 podStartE2EDuration="1.126230818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:42.123620132 +0000 UTC m=+5556.553489994" watchObservedRunningTime="2026-11-03 00:20:42.126230818 +0000 UTC m=+5556.556100673" Nov 03 00:20:42.380738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.380645 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:42.382062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.382037 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:42.382227 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:42.382087 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:43.381731 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:43.381693 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.382154 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:43.381765 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:44.381821 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:44.381780 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.382280 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:44.381854 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:45.381843 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:45.381795 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.382476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:45.381885 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:46.381754 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:46.381714 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:46.382081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:46.382048 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:47.212658 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:47.212625 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:20:47.383197 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:47.382642 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:47.383197 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:47.382739 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:47.676809 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:20:47.676745 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:20:48.381610 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:48.381573 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body= Nov 03 00:20:48.381809 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:48.381636 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" Nov 03 00:20:49.803681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:49.803646 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:20:50.381742 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:50.381700 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:50.381962 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:50.381774 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:51.423444 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:51.423405 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:51.423951 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:51.423478 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:52.159945 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:52.159905 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:52.160386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:52.160358 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:52.161159 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:52.161140 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" Nov 03 00:20:59.425371 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:20:59.425341 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:21:08.288205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:08.288152 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:21:11.284717 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.284687 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v"] Nov 03 00:21:11.288382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.288362 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368577 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1eab1e-ff5b-4337-8058-c10970f7761e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368614 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368634 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368690 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368787 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368842 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368874 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhth\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-kube-api-access-2zhth\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368903 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.368961 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.368923 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.392091 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.392062 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v"] Nov 03 00:21:11.470399 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470366 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470407 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470448 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhth\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-kube-api-access-2zhth\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470483 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470515 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470814 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470611 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1eab1e-ff5b-4337-8058-c10970f7761e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470814 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470642 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470814 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470665 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470814 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470701 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.470814 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470808 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.471060 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.470853 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.471215 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.471164 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.471313 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.471209 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.471440 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.471419 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1eab1e-ff5b-4337-8058-c10970f7761e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.473303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.473278 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.473497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.473482 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.531985 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.531953 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.532274 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.532252 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhth\" (UniqueName: \"kubernetes.io/projected/0a1eab1e-ff5b-4337-8058-c10970f7761e-kube-api-access-2zhth\") pod \"maas-default-gateway-openshift-default-694db55c87-8fv4v\" (UID: \"0a1eab1e-ff5b-4337-8058-c10970f7761e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:11.601657 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:11.601578 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:12.033286 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.033262 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v"] Nov 03 00:21:12.034948 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:21:12.034920 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1eab1e_ff5b_4337_8058_c10970f7761e.slice/crio-5a59e16bbaf733e80b9513fbcf701ccf9779905e87fa7bc91981a9ff81e994af WatchSource:0}: Error finding container 5a59e16bbaf733e80b9513fbcf701ccf9779905e87fa7bc91981a9ff81e994af: Status 404 returned error can't find the container with id 5a59e16bbaf733e80b9513fbcf701ccf9779905e87fa7bc91981a9ff81e994af Nov 03 00:21:12.036979 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.036942 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:12.037062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.037008 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:12.037062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.037039 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:12.184208 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.184152 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" event={"ID":"0a1eab1e-ff5b-4337-8058-c10970f7761e","Type":"ContainerStarted","Data":"b016813744a201bce6b1c818494ea11c8dcccc1a1f165733a7b2356d9ad0e5dc"} Nov 03 00:21:12.184428 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.184234 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" event={"ID":"0a1eab1e-ff5b-4337-8058-c10970f7761e","Type":"ContainerStarted","Data":"5a59e16bbaf733e80b9513fbcf701ccf9779905e87fa7bc91981a9ff81e994af"} Nov 03 00:21:12.306580 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.306473 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podStartSLOduration=1.306457785 podStartE2EDuration="1.306457785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:12.304618366 +0000 UTC m=+5586.734488240" watchObservedRunningTime="2026-11-03 00:21:12.306457785 +0000 UTC m=+5586.736327642" Nov 03 00:21:12.602362 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:12.602284 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:13.604003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:13.603967 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:13.604435 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:13.604015 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:14.602489 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:14.602458 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:14.602679 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:14.602508 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:15.602260 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:15.602225 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:15.602681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:15.602277 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:16.602319 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:16.602285 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:16.602777 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:16.602342 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:17.590690 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:21:17.590641 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:21:17.602606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:17.602578 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:17.602968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:17.602628 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:17.674535 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:17.674499 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:21:18.602779 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:18.602741 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:18.603361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:18.602811 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:19.606682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:19.606646 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Nov 03 00:21:19.607366 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:19.607314 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Nov 03 00:21:21.611069 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:21.611036 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:21:21.611540 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:21.611107 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:21:22.653326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:22.653274 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:21:22.653872 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:22.653344 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:21:23.537127 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:23.537093 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:23.537514 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:23.537496 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:23.538228 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:23.538207 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" Nov 03 00:21:27.481491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:27.481450 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:28.281933 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:28.281884 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:21:46.263452 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:46.263416 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:21:47.452035 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:47.451976 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:47.603684 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:47.603443 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:47.685714 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:21:47.685678 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:21:48.247194 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:48.246985 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:51.758389 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:51.758344 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:21:55.769738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:21:55.769696 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:22:09.440834 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:09.440803 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:09.637456 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:09.637422 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:09.783156 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:09.783044 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:10.046691 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:10.046603 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf"] Nov 03 00:22:10.096895 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:10.096856 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ec64ed-nfqnf"] Nov 03 00:22:10.214363 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:10.214332 4494 scope.go:122] "RemoveContainer" containerID="dd43e194e6caa983a4436988a6efc2be7cf5ca256e7bc8b64cdf74fcf2016606" Nov 03 00:22:11.692009 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:11.691974 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40ccabe-7e53-421a-984b-89f13b45994e" path="/var/lib/kubelet/pods/f40ccabe-7e53-421a-984b-89f13b45994e/volumes" Nov 03 00:22:17.558094 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:22:17.558058 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:22:22.047329 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:22.047298 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v"] Nov 03 00:22:22.097617 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:22.097584 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bcfa411-4n97v"] Nov 03 00:22:23.692931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:23.692897 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f9f778-17ff-43e2-a14e-68fd8914d1ee" path="/var/lib/kubelet/pods/11f9f778-17ff-43e2-a14e-68fd8914d1ee/volumes" Nov 03 00:22:27.821808 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:27.821770 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:32.078122 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:32.078095 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:22:43.185736 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.185698 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw"] Nov 03 00:22:43.189801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.189777 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.244655 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.244612 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:22:43.295618 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.295584 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:22:43.297665 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.297644 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw"] Nov 03 00:22:43.343519 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.343489 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.343681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.343619 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkb2\" (UniqueName: \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.343728 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.343678 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.444634 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.444548 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkb2\" (UniqueName: \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.444634 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.444599 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.444847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.444655 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.445430 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.445404 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.447163 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.447140 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.506847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.506821 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkb2\" (UniqueName: \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\") pod \"maas-api-revoke-sub-492ca87c-crmcw\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:43.803312 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:43.803215 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:44.176299 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:44.176275 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw"] Nov 03 00:22:44.178026 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:22:44.177998 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49eabd95_61f9_4124_98ef_0f10e89f14bd.slice/crio-3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d WatchSource:0}: Error finding container 3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d: Status 404 returned error can't find the container with id 3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d Nov 03 00:22:44.814907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:44.814808 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" event={"ID":"49eabd95-61f9-4124-98ef-0f10e89f14bd","Type":"ContainerStarted","Data":"8c26c8fad8e76c8d8a7d64153020035b37a932b509a57009582db1785047a19c"} Nov 03 00:22:44.814907 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:44.814856 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" event={"ID":"49eabd95-61f9-4124-98ef-0f10e89f14bd","Type":"ContainerStarted","Data":"3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d"} Nov 03 00:22:44.931688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:44.931630 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" podStartSLOduration=1.931615165 podStartE2EDuration="1.931615165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:44.931318603 +0000 UTC m=+5679.361188467" watchObservedRunningTime="2026-11-03 00:22:44.931615165 +0000 UTC m=+5679.361485028" Nov 03 00:22:45.820878 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:45.820839 4494 generic.go:356] "Generic (PLEG): container finished" podID="49eabd95-61f9-4124-98ef-0f10e89f14bd" containerID="8c26c8fad8e76c8d8a7d64153020035b37a932b509a57009582db1785047a19c" exitCode=0 Nov 03 00:22:45.821335 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:45.820938 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" event={"ID":"49eabd95-61f9-4124-98ef-0f10e89f14bd","Type":"ContainerDied","Data":"8c26c8fad8e76c8d8a7d64153020035b37a932b509a57009582db1785047a19c"} Nov 03 00:22:46.965512 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:46.965482 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:47.091159 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.091074 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\") pod \"49eabd95-61f9-4124-98ef-0f10e89f14bd\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " Nov 03 00:22:47.091348 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.091199 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\") pod \"49eabd95-61f9-4124-98ef-0f10e89f14bd\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " Nov 03 00:22:47.091348 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.091288 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\" (UniqueName: \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\") pod \"49eabd95-61f9-4124-98ef-0f10e89f14bd\" (UID: \"49eabd95-61f9-4124-98ef-0f10e89f14bd\") " Nov 03 00:22:47.091583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.091561 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca" pod "49eabd95-61f9-4124-98ef-0f10e89f14bd" (UID: "49eabd95-61f9-4124-98ef-0f10e89f14bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:22:47.091663 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.091655 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:47.093409 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.093374 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca" pod "49eabd95-61f9-4124-98ef-0f10e89f14bd" (UID: "49eabd95-61f9-4124-98ef-0f10e89f14bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:22:47.093661 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.093633 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2" pod "49eabd95-61f9-4124-98ef-0f10e89f14bd" (UID: "49eabd95-61f9-4124-98ef-0f10e89f14bd"). InnerVolumeSpecName "kube-api-access-hbkb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:47.192939 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.192901 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbkb2\" (UniqueName: \"kubernetes.io/projected/49eabd95-61f9-4124-98ef-0f10e89f14bd-kube-api-access-hbkb2\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:47.192939 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.192933 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49eabd95-61f9-4124-98ef-0f10e89f14bd-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:47.559631 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:22:47.559594 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:22:47.827725 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.827631 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:47.832227 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.832202 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" Nov 03 00:22:47.832380 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.832214 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw" event={"ID":"49eabd95-61f9-4124-98ef-0f10e89f14bd","Type":"ContainerDied","Data":"3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d"} Nov 03 00:22:47.832380 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:47.832291 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f31ba9393fbf156438b92506c368adcf6a9b1bc38389262e65595b4a29a052d" Nov 03 00:22:56.095897 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:56.095863 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:22:56.195251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:56.195213 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:22:56.344563 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:22:56.344528 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:01.716981 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.716940 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8"] Nov 03 00:23:01.717783 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.717759 4494 cpu_manager.go:414] "containerMap: removing container" podUID="49eabd95-61f9-4124-98ef-0f10e89f14bd" containerName="revoke-keys" Nov 03 00:23:01.717899 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.717784 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49eabd95-61f9-4124-98ef-0f10e89f14bd" containerName="revoke-keys" Nov 03 00:23:01.717998 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.717904 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="49eabd95-61f9-4124-98ef-0f10e89f14bd" containerName="revoke-keys" Nov 03 00:23:01.721599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.721576 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.738786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.738725 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.738960 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.738859 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq25g\" (UniqueName: \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.738960 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.738931 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.778347 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.778307 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8"] Nov 03 00:23:01.825146 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.825109 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:01.825460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.825412 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:23:01.851088 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.851039 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.852067 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.852041 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq25g\" (UniqueName: \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.852371 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.852343 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.853247 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.853222 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.856594 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.856558 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:01.915999 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:01.915930 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq25g\" (UniqueName: \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\") pod \"maas-api-revoke-sub-b60cf33d-tchg8\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:02.041060 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:02.040890 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:02.467203 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:02.451058 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8"] Nov 03 00:23:02.931198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:02.930716 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" event={"ID":"c12191bb-9409-42ca-b4bc-591a56b6c276","Type":"ContainerStarted","Data":"2c2a413a1fd3e12924c726acb43d03f941ba6353c1a1a0feea1d80e5ce74a49c"} Nov 03 00:23:03.953765 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:03.953728 4494 generic.go:356] "Generic (PLEG): container finished" podID="c12191bb-9409-42ca-b4bc-591a56b6c276" containerID="62335a6aa04bf795022577842cc94fba1c5543f56db0a4150f9b8f4285a4c999" exitCode=0 Nov 03 00:23:03.954480 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:03.954452 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" event={"ID":"c12191bb-9409-42ca-b4bc-591a56b6c276","Type":"ContainerDied","Data":"62335a6aa04bf795022577842cc94fba1c5543f56db0a4150f9b8f4285a4c999"} Nov 03 00:23:05.093490 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.093460 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:05.143053 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.143019 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\") pod \"c12191bb-9409-42ca-b4bc-591a56b6c276\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " Nov 03 00:23:05.143278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.143097 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\" (UniqueName: \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\") pod \"c12191bb-9409-42ca-b4bc-591a56b6c276\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " Nov 03 00:23:05.143278 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.143159 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\") pod \"c12191bb-9409-42ca-b4bc-591a56b6c276\" (UID: \"c12191bb-9409-42ca-b4bc-591a56b6c276\") " Nov 03 00:23:05.143411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.143374 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca" pod "c12191bb-9409-42ca-b4bc-591a56b6c276" (UID: "c12191bb-9409-42ca-b4bc-591a56b6c276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:05.143550 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.143533 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:05.145393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.145373 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g" pod "c12191bb-9409-42ca-b4bc-591a56b6c276" (UID: "c12191bb-9409-42ca-b4bc-591a56b6c276"). InnerVolumeSpecName "kube-api-access-nq25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:05.145477 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.145414 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca" pod "c12191bb-9409-42ca-b4bc-591a56b6c276" (UID: "c12191bb-9409-42ca-b4bc-591a56b6c276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:05.244733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.244651 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq25g\" (UniqueName: \"kubernetes.io/projected/c12191bb-9409-42ca-b4bc-591a56b6c276-kube-api-access-nq25g\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:05.244733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.244681 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12191bb-9409-42ca-b4bc-591a56b6c276-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:05.965578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.965488 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" event={"ID":"c12191bb-9409-42ca-b4bc-591a56b6c276","Type":"ContainerDied","Data":"2c2a413a1fd3e12924c726acb43d03f941ba6353c1a1a0feea1d80e5ce74a49c"} Nov 03 00:23:05.965578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.965525 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8" Nov 03 00:23:05.965578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:05.965538 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c2a413a1fd3e12924c726acb43d03f941ba6353c1a1a0feea1d80e5ce74a49c" Nov 03 00:23:07.191758 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:07.191725 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:23:10.235270 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:10.235237 4494 scope.go:122] "RemoveContainer" containerID="99a047ee371557ad893365a3827deeb9425b28395f90b964c4bc4012fab1225c" Nov 03 00:23:14.200895 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.200858 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:14.707908 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.707875 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:23:14.708343 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.708328 4494 cpu_manager.go:414] "containerMap: removing container" podUID="c12191bb-9409-42ca-b4bc-591a56b6c276" containerName="revoke-keys" Nov 03 00:23:14.708401 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.708344 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12191bb-9409-42ca-b4bc-591a56b6c276" containerName="revoke-keys" Nov 03 00:23:14.708443 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.708431 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12191bb-9409-42ca-b4bc-591a56b6c276" containerName="revoke-keys" Nov 03 00:23:14.713019 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.713002 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.769408 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.769378 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ee0b7a-openshift-default-dockercfg-747rp\"" Nov 03 00:23:14.802570 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.802533 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:14.822819 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.822778 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:23:14.846118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846077 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846306 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846249 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846306 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846299 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846415 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846355 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846497 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846473 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846546 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846534 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846568 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846609 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.846738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.846718 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6gr\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.947926 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.947889 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948124 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.947940 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6gr\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948136 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948271 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948378 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948316 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948378 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948349 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948386 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948432 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948467 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948493 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948610 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948794 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948734 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.948878 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.948857 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.949142 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.949124 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.950736 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.950714 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:14.950937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:14.950921 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:15.010044 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.009972 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:15.010237 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.010162 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6gr\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\") pod \"e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:15.026135 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.025865 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:15.077857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.077821 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:15.461052 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.461028 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:23:15.462457 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:23:15.462427 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e283e3_04dc_469f_ae82_18fc3dfda4b2.slice/crio-efd422c0b4ab3aa8438e397f1834b24d3ef4e24b3adc1f37db7bf3114690db05 WatchSource:0}: Error finding container efd422c0b4ab3aa8438e397f1834b24d3ef4e24b3adc1f37db7bf3114690db05: Status 404 returned error can't find the container with id efd422c0b4ab3aa8438e397f1834b24d3ef4e24b3adc1f37db7bf3114690db05 Nov 03 00:23:15.464599 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.464570 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:15.464663 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.464629 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:15.464663 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:15.464660 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:16.018490 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:16.018455 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerStarted","Data":"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5"} Nov 03 00:23:16.018688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:16.018512 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerStarted","Data":"efd422c0b4ab3aa8438e397f1834b24d3ef4e24b3adc1f37db7bf3114690db05"} Nov 03 00:23:16.026882 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:16.026858 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:16.031515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:16.031497 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:16.141984 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:16.141935 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podStartSLOduration=2.141919595 podStartE2EDuration="2.141919595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:16.140265794 +0000 UTC m=+5710.570135657" watchObservedRunningTime="2026-11-03 00:23:16.141919595 +0000 UTC m=+5710.571789435" Nov 03 00:23:17.023904 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:17.023874 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:17.024933 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:17.024913 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:17.558739 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:23:17.558644 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:23:19.097692 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:19.097616 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:19.348257 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:19.347788 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:19.527413 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:19.527380 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:19.984482 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:19.984447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:22.396357 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:22.396318 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:22.568756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:22.568724 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:25.503950 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.503917 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:23:25.507740 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.507715 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.544436 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.544401 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.544614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.544449 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.544614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.544465 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd27f\" (UniqueName: \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.544614 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.544548 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.563302 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.563267 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ee0b7a-serving-cert\"" Nov 03 00:23:25.563534 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.563375 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:23:25.566115 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.566090 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:23:25.613623 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.613596 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:23:25.645885 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.645844 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.646087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.645931 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.646087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.645961 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd27f\" (UniqueName: \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.646087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.645991 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.646333 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:23:25.646099 4494 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-serving-cert: secret "maas-api-e2e-worker-single-main-ee0b7a-serving-cert" not found Nov 03 00:23:25.646333 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:23:25.646198 4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls podName:8ce6330a-a7e8-4544-85ba-9a95650aad54 nodeName:}" failed. No retries permitted until 2026-11-03 00:23:26.146157838 +0000 UTC m=+5720.576027694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls") pod "maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" (UID: "8ce6330a-a7e8-4544-85ba-9a95650aad54") : secret "maas-api-e2e-worker-single-main-ee0b7a-serving-cert" not found Nov 03 00:23:25.646738 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.646712 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.649265 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.649237 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.703720 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.703680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:25.708769 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.708741 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd27f\" (UniqueName: \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:25.900893 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:25.900771 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:26.050617 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:26.050585 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:26.151671 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:26.151588 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:26.154239 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:26.154220 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:26.423300 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:26.423267 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:26.802094 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:26.802065 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:23:26.803875 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:23:26.803843 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce6330a_a7e8_4544_85ba_9a95650aad54.slice/crio-a5a9b9c74f29084a80cd5b03d9315e4f58d35a01969ab52a0d908316f02c88a9 WatchSource:0}: Error finding container a5a9b9c74f29084a80cd5b03d9315e4f58d35a01969ab52a0d908316f02c88a9: Status 404 returned error can't find the container with id a5a9b9c74f29084a80cd5b03d9315e4f58d35a01969ab52a0d908316f02c88a9 Nov 03 00:23:27.088962 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:27.088877 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" event={"ID":"8ce6330a-a7e8-4544-85ba-9a95650aad54","Type":"ContainerStarted","Data":"a5a9b9c74f29084a80cd5b03d9315e4f58d35a01969ab52a0d908316f02c88a9"} Nov 03 00:23:28.096743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:28.096711 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" event={"ID":"8ce6330a-a7e8-4544-85ba-9a95650aad54","Type":"ContainerStarted","Data":"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f"} Nov 03 00:23:28.097221 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:28.096851 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:28.162107 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:28.162040 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:28.224799 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:28.224745 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" podStartSLOduration=2.370402574 podStartE2EDuration="3.224728999s" totalImagesPullingTime="854.326425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:28.222804575 +0000 UTC m=+5722.652674440" watchObservedRunningTime="2026-11-03 00:23:28.224728999 +0000 UTC m=+5722.654598865" Nov 03 00:23:30.758367 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:30.758312 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:33.680763 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:33.680727 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:23:34.107031 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:34.106950 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:23:35.781621 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:35.781584 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:40.812351 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:40.811246 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:43.195020 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.194980 4494 generic.go:356] "Generic (PLEG): container finished" podID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerID="5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5" exitCode=0 Nov 03 00:23:43.195688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.195029 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerDied","Data":"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5"} Nov 03 00:23:43.195688 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.195446 4494 scope.go:122] "RemoveContainer" containerID="5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5" Nov 03 00:23:43.196701 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.196669 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:43.196827 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.196743 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:43.196827 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.196775 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:43.877457 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:43.877412 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:44.201538 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:44.201501 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerStarted","Data":"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6"} Nov 03 00:23:44.467903 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:44.467821 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:44.614523 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:44.614489 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:44.765761 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:44.765691 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:45.027562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:45.027479 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:45.028579 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:45.028551 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 03 00:23:45.028718 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:45.028610 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 03 00:23:46.027372 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:46.027339 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 03 00:23:46.027864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:46.027397 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 03 00:23:46.089254 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:46.089223 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:47.024681 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:47.024645 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:47.027458 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:47.027438 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 03 00:23:47.027758 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:47.027494 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 03 00:23:47.587215 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:23:47.587125 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:23:47.902812 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:47.902719 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:48.027463 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:48.027433 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 03 00:23:48.027887 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:48.027494 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 03 00:23:48.197185 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:48.197140 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:49.027013 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:49.026971 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body= Nov 03 00:23:49.027242 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:49.027048 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" Nov 03 00:23:50.031128 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:50.031097 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:50.032222 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:50.032203 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:23:51.097976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:51.097943 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:23:52.553929 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:52.553882 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:23:56.054555 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:23:56.054518 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:00.976956 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:00.976914 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:05.936605 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:05.936567 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:10.259179 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:10.259148 4494 scope.go:122] "RemoveContainer" containerID="39b9fdf943ec897d7908f418ec4a1a869e7d060bef9aea4728609ef50ddfb63c" Nov 03 00:24:10.840205 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:10.840152 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:15.753400 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:15.753361 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:17.559586 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:24:17.559540 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:24:20.650032 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:20.649993 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:23.001580 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:23.001544 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:24:25.563491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:25.563461 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:28.513106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:28.513074 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:30.446190 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:30.446152 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:35.441512 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:35.441471 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:40.350204 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:40.350144 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:44.050880 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:44.050847 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw"] Nov 03 00:24:44.102464 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:44.102426 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2fmkw"] Nov 03 00:24:45.351606 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:45.351574 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:45.694940 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:45.694898 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d736409-4c28-4871-927a-8c6d4e75c808" path="/var/lib/kubelet/pods/7d736409-4c28-4871-927a-8c6d4e75c808/volumes" Nov 03 00:24:47.558292 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:24:47.558192 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:24:48.543817 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:48.543781 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:48.947475 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:48.947417 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:49.427629 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:49.427576 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:50.251012 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:50.250952 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:50.799955 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:50.799924 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:24:54.066740 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:54.066703 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:55.283158 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:55.283123 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:55.427637 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:55.427596 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:56.280507 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:56.280468 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:24:56.427188 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:24:56.427131 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:00.545081 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:00.544986 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:00.969982 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:00.969931 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:04.795460 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:04.795420 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:25:04.795970 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:04.795660 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" podUID="8ce6330a-a7e8-4544-85ba-9a95650aad54" containerName="maas-api" containerID="cri-o://b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f" gracePeriod=30 Nov 03 00:25:04.899432 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:04.899362 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:06.059857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.059833 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:25:06.154304 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154224 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") pod \"8ce6330a-a7e8-4544-85ba-9a95650aad54\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " Nov 03 00:25:06.154479 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154334 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\" (UniqueName: \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\") pod \"8ce6330a-a7e8-4544-85ba-9a95650aad54\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " Nov 03 00:25:06.154479 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154438 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\") pod \"8ce6330a-a7e8-4544-85ba-9a95650aad54\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " Nov 03 00:25:06.154600 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154572 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\") pod \"8ce6330a-a7e8-4544-85ba-9a95650aad54\" (UID: \"8ce6330a-a7e8-4544-85ba-9a95650aad54\") " Nov 03 00:25:06.154795 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154765 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca" pod "8ce6330a-a7e8-4544-85ba-9a95650aad54" (UID: "8ce6330a-a7e8-4544-85ba-9a95650aad54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:06.155086 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.154934 4494 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce6330a-a7e8-4544-85ba-9a95650aad54-openshift-service-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.156674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.156648 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f" pod "8ce6330a-a7e8-4544-85ba-9a95650aad54" (UID: "8ce6330a-a7e8-4544-85ba-9a95650aad54"). InnerVolumeSpecName "kube-api-access-qd27f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:06.156674 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.156661 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs" pod "8ce6330a-a7e8-4544-85ba-9a95650aad54" (UID: "8ce6330a-a7e8-4544-85ba-9a95650aad54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:06.156832 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.156678 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls" pod "8ce6330a-a7e8-4544-85ba-9a95650aad54" (UID: "8ce6330a-a7e8-4544-85ba-9a95650aad54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:06.256368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.256334 4494 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-metrics-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.256368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.256361 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ce6330a-a7e8-4544-85ba-9a95650aad54-maas-api-tls\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.256368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.256372 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd27f\" (UniqueName: \"kubernetes.io/projected/8ce6330a-a7e8-4544-85ba-9a95650aad54-kube-api-access-qd27f\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.643533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.643500 4494 generic.go:356] "Generic (PLEG): container finished" podID="8ce6330a-a7e8-4544-85ba-9a95650aad54" containerID="b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f" exitCode=0 Nov 03 00:25:06.643733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.643564 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" Nov 03 00:25:06.643733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.643591 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" event={"ID":"8ce6330a-a7e8-4544-85ba-9a95650aad54","Type":"ContainerDied","Data":"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f"} Nov 03 00:25:06.643733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.643663 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt" event={"ID":"8ce6330a-a7e8-4544-85ba-9a95650aad54","Type":"ContainerDied","Data":"a5a9b9c74f29084a80cd5b03d9315e4f58d35a01969ab52a0d908316f02c88a9"} Nov 03 00:25:06.643733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.643686 4494 scope.go:122] "RemoveContainer" containerID="b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f" Nov 03 00:25:06.653880 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.653864 4494 scope.go:122] "RemoveContainer" containerID="b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f" Nov 03 00:25:06.654229 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:25:06.654203 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f\": container with ID starting with b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f not found: ID does not exist" containerID="b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f" Nov 03 00:25:06.654290 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.654242 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f"} err="failed to get container status \"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f\": rpc error: code = NotFound desc = could not find container \"b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f\": container with ID starting with b937be71d97ea4ac4532d838f3977ee6f8eb34ab24c458d827a21f26fcf6617f not found: ID does not exist" Nov 03 00:25:06.809953 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.809911 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:25:06.860619 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:06.860586 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0b7a-657477f886-fwdbt"] Nov 03 00:25:07.093440 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:07.093407 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:07.246317 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:07.246284 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:07.695271 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:07.695238 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce6330a-a7e8-4544-85ba-9a95650aad54" path="/var/lib/kubelet/pods/8ce6330a-a7e8-4544-85ba-9a95650aad54/volumes" Nov 03 00:25:09.584251 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:09.584219 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:25:10.280112 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:10.280083 4494 scope.go:122] "RemoveContainer" containerID="564d5ab783661453ffd572c5dc02a06b2f898f28fbcee43914b48c23c9dce3ce" Nov 03 00:25:15.418332 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:15.418296 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:17.560560 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:25:17.560521 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:25:25.880569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:25.880536 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:28.855579 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:28.855545 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:31.104694 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:31.104663 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:31.490666 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:31.490631 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:32.661472 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:32.661438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:33.270941 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:33.270900 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:25:33.271183 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:33.271138 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" containerID="cri-o://cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6" gracePeriod=30 Nov 03 00:25:33.279047 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:33.279025 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:35.034418 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:35.034377 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:25:35.034837 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:35.034460 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:25:37.282320 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:37.282283 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:37.429289 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:37.429258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:38.523141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.523117 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:25:38.601252 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601141 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601252 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601246 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601292 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601325 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601450 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601510 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601584 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601537 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601593 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601629 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\") pod \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\" (UID: \"f4e283e3-04dc-469f-ae82-18fc3dfda4b2\") " Nov 03 00:25:38.601850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601707 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:38.601850 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.601764 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:38.602096 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.602026 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.602096 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.602054 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.602356 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.602297 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:38.602467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.602365 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:38.602622 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.602594 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:38.603724 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.603698 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:38.603944 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.603920 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "kube-api-access-4c6gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:38.604535 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.604514 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:25:38.604535 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.604528 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token" pod "f4e283e3-04dc-469f-ae82-18fc3dfda4b2" (UID: "f4e283e3-04dc-469f-ae82-18fc3dfda4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:38.702699 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702668 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702699 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702699 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702712 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6gr\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-kube-api-access-4c6gr\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702727 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702738 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702751 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.702957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.702762 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4e283e3-04dc-469f-ae82-18fc3dfda4b2-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:38.807326 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.807294 4494 generic.go:356] "Generic (PLEG): container finished" podID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerID="cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6" exitCode=0 Nov 03 00:25:38.807504 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.807346 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" Nov 03 00:25:38.807504 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.807348 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerDied","Data":"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6"} Nov 03 00:25:38.807504 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.807432 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk" event={"ID":"f4e283e3-04dc-469f-ae82-18fc3dfda4b2","Type":"ContainerDied","Data":"efd422c0b4ab3aa8438e397f1834b24d3ef4e24b3adc1f37db7bf3114690db05"} Nov 03 00:25:38.807504 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.807455 4494 scope.go:122] "RemoveContainer" containerID="cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6" Nov 03 00:25:38.817022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.817004 4494 scope.go:122] "RemoveContainer" containerID="5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5" Nov 03 00:25:38.825064 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.825047 4494 scope.go:122] "RemoveContainer" containerID="cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6" Nov 03 00:25:38.825356 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:25:38.825337 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6\": container with ID starting with cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6 not found: ID does not exist" containerID="cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6" Nov 03 00:25:38.825445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.825365 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6"} err="failed to get container status \"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6\": rpc error: code = NotFound desc = could not find container \"cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6\": container with ID starting with cfe1f0e2195e23b4711fa957442436ef50c18caaa04a2838b5819a23d2cd72c6 not found: ID does not exist" Nov 03 00:25:38.825445 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.825383 4494 scope.go:122] "RemoveContainer" containerID="5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5" Nov 03 00:25:38.825648 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:25:38.825631 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5\": container with ID starting with 5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5 not found: ID does not exist" containerID="5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5" Nov 03 00:25:38.825694 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.825657 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5"} err="failed to get container status \"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5\": rpc error: code = NotFound desc = could not find container \"5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5\": container with ID starting with 5f0f81cce42a7bcbaa7383e9dd31fe8523c3897b6864a0cc65ad96dcd32beac5 not found: ID does not exist" Nov 03 00:25:38.975089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.975054 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:25:38.978877 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:38.978850 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0b7a-openshift-default-766957c5df4vjmk"] Nov 03 00:25:39.692293 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:39.692260 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" path="/var/lib/kubelet/pods/f4e283e3-04dc-469f-ae82-18fc3dfda4b2/volumes" Nov 03 00:25:47.559097 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:25:47.559060 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:25:48.895414 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:48.895372 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:52.145991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:52.145958 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:25:56.701795 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:25:56.701762 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:26:10.305592 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:10.305560 4494 scope.go:122] "RemoveContainer" containerID="254e969671221b13b222e5b2c0e8ce7dfd3f8babe274ceff6bd7005945e5dcc0" Nov 03 00:26:16.682465 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:16.682431 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:17.558399 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:26:17.558316 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:26:23.257285 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:23.257251 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:27.977603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:27.977565 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:26:29.225616 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:29.225573 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:29.370966 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:29.370921 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:47.558357 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:26:47.558313 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:26:49.243495 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:49.243446 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:26:58.575912 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:26:58.575874 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:27:10.186601 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:10.186570 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:14.093867 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:14.093833 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:27:17.581860 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:27:17.581801 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:27:26.625536 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:26.625487 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:29.574322 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:29.574279 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:31.175198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:31.175136 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:27:43.596360 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:43.596328 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:27:47.045476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:47.045441 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw"] Nov 03 00:27:47.095642 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:47.095609 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-492ca87c-crmcw"] Nov 03 00:27:47.559876 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:27:47.559774 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:27:47.692001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:47.691956 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49eabd95-61f9-4124-98ef-0f10e89f14bd" path="/var/lib/kubelet/pods/49eabd95-61f9-4124-98ef-0f10e89f14bd/volumes" Nov 03 00:27:49.573235 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:49.573159 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:27:54.655134 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:27:54.655102 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:28:06.047595 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:06.047558 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8"] Nov 03 00:28:06.098144 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:06.098108 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60cf33d-tchg8"] Nov 03 00:28:07.692708 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:07.692670 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12191bb-9409-42ca-b4bc-591a56b6c276" path="/var/lib/kubelet/pods/c12191bb-9409-42ca-b4bc-591a56b6c276/volumes" Nov 03 00:28:09.067720 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:09.067688 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:28:17.558614 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:28:17.558574 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:28:29.925330 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:29.925298 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:39.934712 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:39.934672 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:40.079474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:40.079439 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:47.557934 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:28:47.557884 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:28:49.989926 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:49.989873 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:56.634121 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:56.634059 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:57.093838 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:57.093807 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:28:59.831484 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:28:59.831446 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:01.564656 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:01.564621 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:29:10.343709 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:10.343678 4494 scope.go:122] "RemoveContainer" containerID="62335a6aa04bf795022577842cc94fba1c5543f56db0a4150f9b8f4285a4c999" Nov 03 00:29:10.352344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:10.352323 4494 scope.go:122] "RemoveContainer" containerID="8c26c8fad8e76c8d8a7d64153020035b37a932b509a57009582db1785047a19c" Nov 03 00:29:17.558528 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:29:17.558435 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:29:30.332693 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:30.332655 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:36.268295 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:36.268258 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:29:47.560383 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:29:47.560339 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:29:50.292214 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:29:50.292158 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:17.559340 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:30:17.559296 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:30:30.623033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:30.622991 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:47.559223 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:30:47.559122 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:30:50.074390 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:50.074357 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:50.223312 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:50.223279 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:50.631001 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:50.630960 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:56.825482 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:56.825443 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:56.969015 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:56.968978 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:57.312777 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:57.312740 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:30:59.852411 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:30:59.852379 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:31:10.548839 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:10.548808 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:31:17.561498 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:31:17.561455 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:31:28.990933 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:28.990894 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:31:30.968459 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:30.968400 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:39.653791 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:39.653748 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:31:47.558496 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:31:47.558449 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:31:50.988769 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:50.988728 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:31:57.759246 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:31:57.759212 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:32:10.959115 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:10.959074 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:32:17.559232 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:32:17.559128 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:32:31.319968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:31.319925 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:41.863506 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:41.863477 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:32:47.558523 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:32:47.558476 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:32:51.308417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:51.308381 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:32:52.490050 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:52.490013 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:32:58.044783 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:32:58.044700 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:04.971368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:04.971329 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:33:17.101818 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:17.101780 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:17.558472 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:33:17.558433 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:33:27.369679 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:27.369643 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:33:31.659676 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:31.659640 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:36.882572 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:36.882531 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:33:45.376389 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:45.376354 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:33:47.558964 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:33:47.558818 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:33:51.736468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:51.736426 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:33:54.588340 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:54.588297 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:33:54.588977 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:54.588945 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:33:57.787635 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:33:57.787606 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:06.301123 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:06.301079 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:15.891911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:15.891875 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:17.558822 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:34:17.558775 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:34:24.567396 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:24.567363 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:32.009344 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:32.009305 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:34:37.972400 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:37.972368 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:47.558882 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:34:47.558841 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:34:48.872511 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:48.872476 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:34:52.027471 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:34:52.027441 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:02.377901 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:02.377857 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:35:12.671179 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:12.671134 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:35:17.559429 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:35:17.559328 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:35:24.384848 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:24.384814 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:35:28.686526 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:28.686491 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:32.350992 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:32.350953 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:45.974973 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:45.974936 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:35:47.750202 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:35:47.749832 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:35:52.369339 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:52.369291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:35:56.662545 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:35:56.662508 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:36:15.288782 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:15.288747 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:36:16.760835 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:16.760762 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:17.305756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.305711 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:36:17.306747 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.306717 4494 cpu_manager.go:414] "containerMap: removing container" podUID="8ce6330a-a7e8-4544-85ba-9a95650aad54" containerName="maas-api" Nov 03 00:36:17.306927 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.306911 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce6330a-a7e8-4544-85ba-9a95650aad54" containerName="maas-api" Nov 03 00:36:17.307044 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.307031 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:36:17.307147 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.307133 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:36:17.307404 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.307388 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce6330a-a7e8-4544-85ba-9a95650aad54" containerName="maas-api" Nov 03 00:36:17.307515 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.307503 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:36:17.307618 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.307607 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:36:17.311500 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.311478 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.367309 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.367280 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-649f61-openshift-default-dockercfg-dgqqc\"" Nov 03 00:36:17.429403 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.429364 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.437279 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.444945 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446567 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446641 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446675 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjrg\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446704 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446735 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446820 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446848 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.447148 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.446932 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.557776 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.557840 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjrg\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.557870 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.557902 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.557999 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.558028 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.558109 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.558162 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.558227 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.558698 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.559429 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.560019 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.590282 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.560310 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.614392 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.614306 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.640220 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.640159 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.650925 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.647257 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.651683 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.651625 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.664277 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.656827 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjrg\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\") pod \"e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:17.729487 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:36:17.729435 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:36:17.871861 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.871772 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:17.944067 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:17.944028 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:18.372745 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.372722 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:36:18.374119 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:36:18.374092 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf02a776_6248_4682_87cf_d8b4987eb731.slice/crio-b3a389cf02d60f145987b22e2dc3510fd35607d4ef808741d12e9e8b561f5202 WatchSource:0}: Error finding container b3a389cf02d60f145987b22e2dc3510fd35607d4ef808741d12e9e8b561f5202: Status 404 returned error can't find the container with id b3a389cf02d60f145987b22e2dc3510fd35607d4ef808741d12e9e8b561f5202 Nov 03 00:36:18.376430 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.376389 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:18.376529 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.376469 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:18.376529 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.376520 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:18.970577 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.970537 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerStarted","Data":"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a"} Nov 03 00:36:18.970986 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:18.970628 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerStarted","Data":"b3a389cf02d60f145987b22e2dc3510fd35607d4ef808741d12e9e8b561f5202"} Nov 03 00:36:19.091857 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:19.091805 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podStartSLOduration=2.091792751 podStartE2EDuration="2.091792751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:19.091147894 +0000 UTC m=+6493.521017756" watchObservedRunningTime="2026-11-03 00:36:19.091792751 +0000 UTC m=+6493.521662612" Nov 03 00:36:19.945070 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:19.945031 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:19.949952 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:19.949927 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:19.975315 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:19.975289 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:19.976279 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:19.976261 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:21.975801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:21.975772 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:22.134580 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:22.134542 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:22.294361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:22.294278 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:25.164003 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:25.163966 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:25.338446 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:25.338404 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:28.491741 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:28.491706 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:28.654583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:28.654547 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:28.799901 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:28.799820 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:29.054561 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:29.054471 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:36:32.700499 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:32.700465 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:33.564305 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:33.564270 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:38.591319 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:38.591280 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:40.863132 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:40.863094 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:36:43.551088 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:43.551054 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:46.106339 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.106303 4494 generic.go:356] "Generic (PLEG): container finished" podID="af02a776-6248-4682-87cf-d8b4987eb731" containerID="86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a" exitCode=0 Nov 03 00:36:46.106794 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.106377 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerDied","Data":"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a"} Nov 03 00:36:46.106794 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.106765 4494 scope.go:122] "RemoveContainer" containerID="86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a" Nov 03 00:36:46.107972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.107948 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:46.108028 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.108014 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:46.108062 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.108046 4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:46.790417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:46.790384 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:47.112664 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.112576 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerStarted","Data":"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94"} Nov 03 00:36:47.462722 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.462682 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:47.585495 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:36:47.585405 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:36:47.617250 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.617209 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:47.946572 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.946539 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:47.950494 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.950463 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 03 00:36:47.950754 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:47.950726 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 03 00:36:48.950733 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:48.950695 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 03 00:36:48.951391 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:48.951359 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 03 00:36:48.995492 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:48.995447 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:49.947678 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:49.947637 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 03 00:36:49.947991 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:49.947957 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 03 00:36:49.975837 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:49.975784 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:50.396000 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:50.395917 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:50.953198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:50.952609 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 03 00:36:50.953198 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:50.952687 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 03 00:36:51.955204 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:51.954193 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" start-of-body= Nov 03 00:36:51.955204 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:51.954292 4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.61:15021/healthz/ready\": dial tcp 10.133.1.61:15021: connect: connection refused" Nov 03 00:36:52.705491 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:52.705455 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:52.949032 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:52.948998 4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:52.949909 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:52.949881 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:36:53.947311 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:53.947275 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:36:55.178297 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:55.178261 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:36:58.930022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:36:58.929988 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:03.849604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:03.849569 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:08.751724 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:08.751687 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:10.511391 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:10.511356 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:13.658906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:13.658869 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:17.558001 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:37:17.557956 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:37:18.553578 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:18.553545 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:20.694684 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:20.694649 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:37:23.461275 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:23.461238 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:28.450786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:28.450750 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:33.046024 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:33.045988 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:34.415769 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.415719 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:34.932620 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.932589 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v"] Nov 03 00:37:34.933038 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.933026 4494 cpu_manager.go:414] "containerMap: removing container" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:37:34.933090 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.933040 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e283e3-04dc-469f-ae82-18fc3dfda4b2" containerName="istio-proxy" Nov 03 00:37:34.935963 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.935947 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:34.964166 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.964140 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s6n\" (UniqueName: \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:34.964312 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.964277 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:34.964366 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.964305 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:34.991227 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:34.991193 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:37:35.041633 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.041603 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:37:35.043645 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.043625 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v"] Nov 03 00:37:35.065518 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.065492 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.065661 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.065529 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.065706 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.065678 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s6n\" (UniqueName: \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.066229 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.066202 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.068070 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.068049 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.129393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.129368 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s6n\" (UniqueName: \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\") pod \"maas-api-revoke-sub-62dbf5c4-tb25v\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.248014 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.247930 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:35.625743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:35.625717 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v"] Nov 03 00:37:35.627249 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:37:35.627224 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be2e585_aa41_497d_8988_89705113b205.slice/crio-31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350 WatchSource:0}: Error finding container 31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350: Status 404 returned error can't find the container with id 31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350 Nov 03 00:37:36.355853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:36.355765 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" event={"ID":"6be2e585-aa41-497d-8988-89705113b205","Type":"ContainerStarted","Data":"d0b588074a4bc89c11c34ea63ac8922faf903befa2c196c146b9f2dfeea27e19"} Nov 03 00:37:36.355853 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:36.355824 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" event={"ID":"6be2e585-aa41-497d-8988-89705113b205","Type":"ContainerStarted","Data":"31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350"} Nov 03 00:37:36.473439 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:36.473380 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" podStartSLOduration=2.47336762 podStartE2EDuration="2.47336762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:36.472860072 +0000 UTC m=+6570.902729927" watchObservedRunningTime="2026-11-03 00:37:36.47336762 +0000 UTC m=+6570.903237472" Nov 03 00:37:37.362067 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:37.362035 4494 generic.go:356] "Generic (PLEG): container finished" podID="6be2e585-aa41-497d-8988-89705113b205" containerID="d0b588074a4bc89c11c34ea63ac8922faf903befa2c196c146b9f2dfeea27e19" exitCode=0 Nov 03 00:37:37.362512 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:37.362115 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" event={"ID":"6be2e585-aa41-497d-8988-89705113b205","Type":"ContainerDied","Data":"d0b588074a4bc89c11c34ea63ac8922faf903befa2c196c146b9f2dfeea27e19"} Nov 03 00:37:38.343457 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.343427 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:38.528930 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.528904 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:38.606700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.606616 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\") pod \"6be2e585-aa41-497d-8988-89705113b205\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " Nov 03 00:37:38.606700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.606656 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\" (UniqueName: \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\") pod \"6be2e585-aa41-497d-8988-89705113b205\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " Nov 03 00:37:38.606700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.606678 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\") pod \"6be2e585-aa41-497d-8988-89705113b205\" (UID: \"6be2e585-aa41-497d-8988-89705113b205\") " Nov 03 00:37:38.607089 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.607064 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca" pod "6be2e585-aa41-497d-8988-89705113b205" (UID: "6be2e585-aa41-497d-8988-89705113b205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:38.608919 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.608896 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca" pod "6be2e585-aa41-497d-8988-89705113b205" (UID: "6be2e585-aa41-497d-8988-89705113b205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:37:38.609015 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.608916 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n" pod "6be2e585-aa41-497d-8988-89705113b205" (UID: "6be2e585-aa41-497d-8988-89705113b205"). InnerVolumeSpecName "kube-api-access-k6s6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:38.708192 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.708149 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2e585-aa41-497d-8988-89705113b205-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:38.708385 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.708206 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s6n\" (UniqueName: \"kubernetes.io/projected/6be2e585-aa41-497d-8988-89705113b205-kube-api-access-k6s6n\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:38.708385 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:38.708224 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2e585-aa41-497d-8988-89705113b205-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:39.375622 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:39.375595 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" Nov 03 00:37:39.375806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:39.375583 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v" event={"ID":"6be2e585-aa41-497d-8988-89705113b205","Type":"ContainerDied","Data":"31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350"} Nov 03 00:37:39.375806 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:39.375720 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31248029f9fe179e41c9d13dd443e6cfb7d6618411c6f1fb5f6701c8d1b1a350" Nov 03 00:37:43.241885 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:43.241841 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:46.573303 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:46.573212 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:47.070375 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:47.070339 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:47.559462 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:37:47.559416 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:37:48.156544 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:48.156507 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:49.793937 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:49.793894 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:37:51.633145 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:51.633084 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:52.718969 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:52.718934 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:52.862308 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:52.862266 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:53.061889 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:53.061805 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:53.206583 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:53.206543 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:53.368040 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:53.367962 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:57.615482 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:57.615438 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:37:58.961338 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:37:58.961303 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:02.194593 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:02.194551 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:38:04.526616 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:04.526580 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:04.674449 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:04.674415 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:17.558607 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:38:17.558523 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:38:28.456382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:28.456351 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:28.947310 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:28.947277 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:29.376477 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:29.376403 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:29.525926 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:29.525885 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:31.694141 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:31.694078 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:31.841775 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:31.841738 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:31.987716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:31.987640 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:33.393481 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:33.393446 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:34.867831 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:34.867735 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:35.013526 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:35.013496 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:36.806072 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:36.805981 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:39.975016 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:39.974973 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:38:41.748237 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:41.748195 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:42.581739 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.581699 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm"] Nov 03 00:38:42.582280 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.582261 4494 cpu_manager.go:414] "containerMap: removing container" podUID="6be2e585-aa41-497d-8988-89705113b205" containerName="revoke-keys" Nov 03 00:38:42.582405 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.582281 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be2e585-aa41-497d-8988-89705113b205" containerName="revoke-keys" Nov 03 00:38:42.582468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.582406 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be2e585-aa41-497d-8988-89705113b205" containerName="revoke-keys" Nov 03 00:38:42.585529 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.585509 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.642682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.642647 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm"] Nov 03 00:38:42.690861 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.690831 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:38:42.691122 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.691103 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gq6rz\"" Nov 03 00:38:42.719438 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.719407 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.719609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.719447 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44gt\" (UniqueName: \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.719609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.719514 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.820891 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.820854 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.820891 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.820895 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44gt\" (UniqueName: \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.821355 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.820932 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.821756 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.821723 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.823476 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.823453 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.882642 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.882574 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44gt\" (UniqueName: \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:42.899382 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:42.899357 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:43.270805 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:43.270775 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm"] Nov 03 00:38:43.272929 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:38:43.272901 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b18983e_6e90_4318_a506_8067475fb3d5.slice/crio-496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c WatchSource:0}: Error finding container 496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c: Status 404 returned error can't find the container with id 496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c Nov 03 00:38:43.737265 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:43.737220 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" event={"ID":"2b18983e-6e90-4318-a506-8067475fb3d5","Type":"ContainerStarted","Data":"6eb718dde49fac3f32895a10a18ce72fdcbce54d72e48cb531aa0578b56bc305"} Nov 03 00:38:43.737464 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:43.737285 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" event={"ID":"2b18983e-6e90-4318-a506-8067475fb3d5","Type":"ContainerStarted","Data":"496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c"} Nov 03 00:38:43.854686 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:43.854587 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" podStartSLOduration=1.8545725819999999 podStartE2EDuration="1.854572582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:43.853790568 +0000 UTC m=+6638.283660430" watchObservedRunningTime="2026-11-03 00:38:43.854572582 +0000 UTC m=+6638.284442466" Nov 03 00:38:44.743569 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:44.743534 4494 generic.go:356] "Generic (PLEG): container finished" podID="2b18983e-6e90-4318-a506-8067475fb3d5" containerID="6eb718dde49fac3f32895a10a18ce72fdcbce54d72e48cb531aa0578b56bc305" exitCode=0 Nov 03 00:38:44.743753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:44.743620 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" event={"ID":"2b18983e-6e90-4318-a506-8067475fb3d5","Type":"ContainerDied","Data":"6eb718dde49fac3f32895a10a18ce72fdcbce54d72e48cb531aa0578b56bc305"} Nov 03 00:38:45.884571 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.884542 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:45.952705 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.952670 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\" (UniqueName: \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\") pod \"2b18983e-6e90-4318-a506-8067475fb3d5\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " Nov 03 00:38:45.952911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.952766 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\") pod \"2b18983e-6e90-4318-a506-8067475fb3d5\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " Nov 03 00:38:45.952911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.952801 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\") pod \"2b18983e-6e90-4318-a506-8067475fb3d5\" (UID: \"2b18983e-6e90-4318-a506-8067475fb3d5\") " Nov 03 00:38:45.953243 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.953220 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca" pod "2b18983e-6e90-4318-a506-8067475fb3d5" (UID: "2b18983e-6e90-4318-a506-8067475fb3d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:45.955022 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.954995 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca" pod "2b18983e-6e90-4318-a506-8067475fb3d5" (UID: "2b18983e-6e90-4318-a506-8067475fb3d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:38:45.955122 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:45.955079 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt" pod "2b18983e-6e90-4318-a506-8067475fb3d5" (UID: "2b18983e-6e90-4318-a506-8067475fb3d5"). InnerVolumeSpecName "kube-api-access-d44gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:46.054386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.054305 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-xks-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:46.054386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.054334 4494 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b18983e-6e90-4318-a506-8067475fb3d5-maas-api-openshift-ca\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:46.054386 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.054345 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44gt\" (UniqueName: \"kubernetes.io/projected/2b18983e-6e90-4318-a506-8067475fb3d5-kube-api-access-d44gt\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:46.600151 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.600097 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:46.754508 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.754474 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" Nov 03 00:38:46.754508 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.754501 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm" event={"ID":"2b18983e-6e90-4318-a506-8067475fb3d5","Type":"ContainerDied","Data":"496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c"} Nov 03 00:38:46.754739 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:46.754548 4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496d24fa7ca3bed6e5bec7b206732cd61d4047841e515c385fc0cb7a56dddd7c" Nov 03 00:38:47.558641 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:38:47.558599 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:38:51.459063 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:51.459028 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:53.459753 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:53.459718 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:38:56.258716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:38:56.258680 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:00.592654 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:00.592575 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:02.088195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:02.088133 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:02.365406 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:02.365317 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:02.751577 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:02.751543 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm"] Nov 03 00:39:02.802461 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:02.802426 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-649f61-vl4xm"] Nov 03 00:39:03.015886 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:03.015787 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:39:03.016125 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:03.016060 4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" containerID="cri-o://18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94" gracePeriod=30 Nov 03 00:39:03.053456 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:03.053427 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:03.699948 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:03.699912 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b18983e-6e90-4318-a506-8067475fb3d5" path="/var/lib/kubelet/pods/2b18983e-6e90-4318-a506-8067475fb3d5/volumes" Nov 03 00:39:07.000032 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:07.000002 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:39:07.951901 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:07.951862 4494 patch_prober.go:29] interesting pod/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:39:07.952098 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:07.951932 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:39:08.275581 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.275548 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:39:08.297788 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.297750 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.297978 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.297797 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.297978 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.297883 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.297988 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298023 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298054 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298090 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298127 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298199 4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\") pod \"af02a776-6248-4682-87cf-d8b4987eb731\" (UID: \"af02a776-6248-4682-87cf-d8b4987eb731\") " Nov 03 00:39:08.298387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298280 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:08.298387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298265 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:08.298387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298318 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:08.298668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298483 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:08.298668 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298551 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:08.298892 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298866 4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-certs\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.298976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298898 4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-credential-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.298976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298913 4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-data\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.298976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298927 4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af02a776-6248-4682-87cf-d8b4987eb731-istiod-ca-cert\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.298976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.298943 4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-workload-socket\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.300267 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.300241 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:39:08.300368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.300324 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:08.300416 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.300383 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:39:08.300757 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.300738 4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg" pod "af02a776-6248-4682-87cf-d8b4987eb731" (UID: "af02a776-6248-4682-87cf-d8b4987eb731"). InnerVolumeSpecName "kube-api-access-zzjrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:08.399604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.399564 4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjrg\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-kube-api-access-zzjrg\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.399604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.399596 4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af02a776-6248-4682-87cf-d8b4987eb731-istio-podinfo\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.399604 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.399609 4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af02a776-6248-4682-87cf-d8b4987eb731-istio-envoy\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.399864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.399621 4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af02a776-6248-4682-87cf-d8b4987eb731-istio-token\") on node \"ip-10-0-135-179.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:08.863705 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.863667 4494 generic.go:356] "Generic (PLEG): container finished" podID="af02a776-6248-4682-87cf-d8b4987eb731" containerID="18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94" exitCode=0 Nov 03 00:39:08.863928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.863717 4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" Nov 03 00:39:08.863928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.863755 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerDied","Data":"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94"} Nov 03 00:39:08.863928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.863819 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6" event={"ID":"af02a776-6248-4682-87cf-d8b4987eb731","Type":"ContainerDied","Data":"b3a389cf02d60f145987b22e2dc3510fd35607d4ef808741d12e9e8b561f5202"} Nov 03 00:39:08.863928 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.863853 4494 scope.go:122] "RemoveContainer" containerID="18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94" Nov 03 00:39:08.874118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.874101 4494 scope.go:122] "RemoveContainer" containerID="86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a" Nov 03 00:39:08.882474 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.882455 4494 scope.go:122] "RemoveContainer" containerID="18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94" Nov 03 00:39:08.882742 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:39:08.882724 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94\": container with ID starting with 18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94 not found: ID does not exist" containerID="18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94" Nov 03 00:39:08.882788 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.882752 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94"} err="failed to get container status \"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94\": rpc error: code = NotFound desc = could not find container \"18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94\": container with ID starting with 18b1557a85b27b0b21e22ee2ea92f31c10b9f7e743d90cdbe5b105b87c6fee94 not found: ID does not exist" Nov 03 00:39:08.882788 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.882771 4494 scope.go:122] "RemoveContainer" containerID="86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a" Nov 03 00:39:08.883026 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:39:08.883008 4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a\": container with ID starting with 86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a not found: ID does not exist" containerID="86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a" Nov 03 00:39:08.883065 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:08.883030 4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a"} err="failed to get container status \"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a\": rpc error: code = NotFound desc = could not find container \"86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a\": container with ID starting with 86266a3c76ab7658bc561b5ecb33e039797be6f54935acda759cd0268d22a78a not found: ID does not exist" Nov 03 00:39:09.031422 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:09.031388 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:39:09.082185 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:09.082147 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-649f61-openshift-default-665b9d89c-btbt6"] Nov 03 00:39:09.692906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:09.692871 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af02a776-6248-4682-87cf-d8b4987eb731" path="/var/lib/kubelet/pods/af02a776-6248-4682-87cf-d8b4987eb731/volumes" Nov 03 00:39:16.409030 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:16.408999 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:16.552739 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:16.552706 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:17.558978 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:39:17.558942 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:39:19.682106 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:19.682071 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:39:32.370549 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:32.370513 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:33.740325 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:33.740291 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:39:47.560108 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:39:47.560017 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:39:53.757136 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:39:53.757101 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:17.558842 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:40:17.558803 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:40:34.093848 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:40:34.093793 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:47.561328 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:40:47.561289 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:40:52.999995 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:40:52.999960 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:40:54.116790 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:40:54.116743 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:05.484617 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:05.484584 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:41:15.258786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:15.258750 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:41:17.561903 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:41:17.561807 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:41:23.658090 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:23.658051 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:41:26.864050 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.863969 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8"] Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864649 4494 cpu_manager.go:414] "containerMap: removing container" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864667 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864690 4494 cpu_manager.go:414] "containerMap: removing container" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864699 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864723 4494 cpu_manager.go:414] "containerMap: removing container" podUID="2b18983e-6e90-4318-a506-8067475fb3d5" containerName="revoke-keys" Nov 03 00:41:26.864792 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864731 4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b18983e-6e90-4318-a506-8067475fb3d5" containerName="revoke-keys" Nov 03 00:41:26.865167 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864843 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.865167 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864858 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="af02a776-6248-4682-87cf-d8b4987eb731" containerName="istio-proxy" Nov 03 00:41:26.865167 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.864870 4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b18983e-6e90-4318-a506-8067475fb3d5" containerName="revoke-keys" Nov 03 00:41:26.868609 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.868589 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:26.958492 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.958452 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzqv\" (UniqueName: \"kubernetes.io/projected/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-kube-api-access-5kzqv\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:26.958716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.958520 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-openshift-service-ca\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:26.958716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.958683 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-maas-api-tls\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:26.958716 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.958713 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-metrics-certs\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:26.972369 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.972337 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:41:26.972369 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.972345 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:41:26.972603 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.972498 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wm4gz\"" Nov 03 00:41:26.975038 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:26.975010 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8"] Nov 03 00:41:27.059665 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.059633 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-maas-api-tls\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.059894 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.059676 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-metrics-certs\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.059894 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.059782 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzqv\" (UniqueName: \"kubernetes.io/projected/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-kube-api-access-5kzqv\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.059894 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.059820 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-openshift-service-ca\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.060484 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.060459 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-openshift-service-ca\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.062375 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.062353 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-metrics-certs\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.062472 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.062396 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-maas-api-tls\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.122227 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.122124 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzqv\" (UniqueName: \"kubernetes.io/projected/30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5-kube-api-access-5kzqv\") pod \"maas-api-69c7f9d898-t8ns8\" (UID: \"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5\") " pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.185244 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.185204 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:27.558215 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.558191 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8"] Nov 03 00:41:27.560208 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:41:27.560155 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cf8a63_5059_4d5a_b70b_ec1c8ef9e4a5.slice/crio-a1607065419f6aee007e4859e636313c420d448b244129c8e85ad96184a95a3c WatchSource:0}: Error finding container a1607065419f6aee007e4859e636313c420d448b244129c8e85ad96184a95a3c: Status 404 returned error can't find the container with id a1607065419f6aee007e4859e636313c420d448b244129c8e85ad96184a95a3c Nov 03 00:41:27.561864 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.561845 4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:41:27.668221 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:27.668160 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" event={"ID":"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5","Type":"ContainerStarted","Data":"a1607065419f6aee007e4859e636313c420d448b244129c8e85ad96184a95a3c"} Nov 03 00:41:28.675080 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:28.674971 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" event={"ID":"30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5","Type":"ContainerStarted","Data":"1e9d27ca3b464a2cd46ee1e2262c9e12ec319797ea5abaf1e081ce792e5259d8"} Nov 03 00:41:28.675080 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:28.675052 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:28.794696 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:28.794643 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" podStartSLOduration=1.941962514 podStartE2EDuration="2.794628972s" totalImagesPullingTime="852.666458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:28.793531143 +0000 UTC m=+6803.223401006" watchObservedRunningTime="2026-11-03 00:41:28.794628972 +0000 UTC m=+6803.224498832" Nov 03 00:41:34.684910 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:34.684877 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-69c7f9d898-t8ns8" Nov 03 00:41:35.703354 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:35.703323 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:39.454005 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:39.453972 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:41:47.562203 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:41:47.562145 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:41:53.960562 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:53.960525 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:54.441184 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:41:54.441138 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:14.984361 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:14.984318 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:42:17.559868 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:42:17.559833 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:42:24.581346 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:24.581308 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:24.582033 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:24.582000 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:27.359906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:27.359866 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:42:34.783815 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:34.783778 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:39.096518 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:39.096478 4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v"] Nov 03 00:42:39.098485 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:39.098458 4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62dbf5c4-tb25v"] Nov 03 00:42:39.692467 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:39.692432 4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be2e585-aa41-497d-8988-89705113b205" path="/var/lib/kubelet/pods/6be2e585-aa41-497d-8988-89705113b205/volumes" Nov 03 00:42:47.560946 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:42:47.560862 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:42:50.578993 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:50.578956 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:42:54.769368 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:54.769329 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:55.890385 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:42:55.890341 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:03.153842 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:03.153806 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:43:17.560633 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:43:17.560596 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:43:35.124331 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:35.124270 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:35.354235 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:35.354196 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:43:45.880957 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:45.880924 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:43:47.661715 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:43:47.661676 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:43:55.232387 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:55.224876 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:58.390424 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:43:58.390391 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:44:10.492468 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:10.492440 4494 scope.go:122] "RemoveContainer" containerID="d0b588074a4bc89c11c34ea63ac8922faf903befa2c196c146b9f2dfeea27e19" Nov 03 00:44:10.606280 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:10.606247 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:44:17.559093 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:44:17.558993 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:44:34.097194 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:34.097147 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:44:35.492188 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:35.492133 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:44.693765 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:44.693732 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:44:47.559029 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:44:47.558990 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:44:54.473883 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:54.473845 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:44:55.514660 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:44:55.514630 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:10.526199 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:10.524799 4494 scope.go:122] "RemoveContainer" containerID="6eb718dde49fac3f32895a10a18ce72fdcbce54d72e48cb531aa0578b56bc305" Nov 03 00:45:14.875974 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:14.875929 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:45:17.695679 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:45:17.695634 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:45:17.812502 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:17.812466 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:24.358974 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:24.358940 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:45:35.833968 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:35.833931 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:47.737484 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:45:47.737377 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:45:55.861439 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:45:55.861406 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:17.561118 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:46:17.561074 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:46:36.175393 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:36.175360 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:47.562136 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:46:47.562089 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:46:53.060725 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.060667 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-k2dss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:53.074496 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.061365 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-k2dss" podUID="80b38964-2c95-47b9-bea3-6a2d26f224cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:46:53.213340 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.213285 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-m8ss5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:53.213682 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.213645 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-m8ss5" podUID="ec3b4496-e1c1-464c-bbfb-d290f8815c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:46:53.245367 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.245322 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-6wkzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:46:53.245744 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:53.245709 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-6wkzh" podUID="bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:46:56.206123 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:46:56.206077 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:17.559636 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:47:17.559544 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:47:24.584826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:24.584330 4494 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8fv4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:47:24.584826 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:24.584410 4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8fv4v" podUID="0a1eab1e-ff5b-4337-8058-c10970f7761e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:47:27.880320 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:27.880282 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:33.106020 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:33.105980 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8dkp"] Nov 03 00:47:36.521215 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:36.521160 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:47.559523 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:47:47.559481 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:47:56.527087 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:47:56.527052 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:00.661449 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:00.661423 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-vhchr_ec7af7fd-654f-45db-a758-9307fb08247e/manager/0.log" Nov 03 00:48:02.928524 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:02.928489 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-69c7f9d898-t8ns8_30cf8a63-5059-4d5a-b70b-ec1c8ef9e4a5/maas-api/0.log" Nov 03 00:48:06.799161 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:06.799129 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g98px_8f3ff83b-6adb-4a10-bc0b-7a7f1247a2e1/postgres/0.log" Nov 03 00:48:10.224092 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:10.224063 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8dkp_5d0a16e7-cd0c-4161-80ea-cdb46ae08b91/limitador/0.log" Nov 03 00:48:11.897709 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:11.897681 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2645m_039a073e-0b49-4ab9-b6f6-79ece1993578/discovery/0.log" Nov 03 00:48:12.471417 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:12.471385 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dbc8865f-r6rbc_edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca/kube-auth-proxy/0.log" Nov 03 00:48:12.699731 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:12.699693 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-6wkzh_bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c/istio-proxy/0.log" Nov 03 00:48:13.300870 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:13.300838 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8fv4v_0a1eab1e-ff5b-4337-8058-c10970f7761e/istio-proxy/0.log" Nov 03 00:48:14.134418 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:14.134388 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-k2dss_80b38964-2c95-47b9-bea3-6a2d26f224cc/istio-proxy/0.log" Nov 03 00:48:14.901462 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:14.901434 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-m8ss5_ec3b4496-e1c1-464c-bbfb-d290f8815c66/istio-proxy/0.log" Nov 03 00:48:17.559108 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:48:17.559074 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:48:17.773582 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:17.773556 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wkpk8_39d7a7ef-1d53-4972-8b3f-8a2d8e499481/main/0.log" Nov 03 00:48:18.049723 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:18.049692 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-bxwpz_1a6e11e2-f926-4ad3-8a2d-9cf289ededc1/main/0.log" Nov 03 00:48:18.273370 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:18.273339 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87q8sr_475f4005-c975-46da-838c-c455dd86f04d/main/0.log" Nov 03 00:48:29.067700 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:29.067669 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:29.226573 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:29.226537 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:36.864481 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:36.864444 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:39.026195 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:39.026148 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k9d98_1555d5cf-a68b-4168-bff4-da817535b17e/global-pull-secret-syncer/0.log" Nov 03 00:48:41.614972 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:41.614940 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" Nov 03 00:48:43.399379 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:43.399352 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-179.us-west-1.compute.internal_b8bcc47f51109dce7950aafef82fe134/haproxy/0.log" Nov 03 00:48:44.826279 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:44.826247 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-vhchr_ec7af7fd-654f-45db-a758-9307fb08247e/manager/0.log" Nov 03 00:48:47.560977 ip-10-0-135-179 kubenswrapper[4494]: E1103 00:48:47.560890 4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rxg4" containerName="extractor" Nov 03 00:48:50.435643 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.435610 4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56"] Nov 03 00:48:50.439160 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.439143 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.492082 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.492050 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"openshift-service-ca.crt\"" Nov 03 00:48:50.492240 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.492115 4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6hpv\"/\"kube-root-ca.crt\"" Nov 03 00:48:50.542718 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.542686 4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6hpv\"/\"default-dockercfg-pwrpp\"" Nov 03 00:48:50.545133 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.545105 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56"] Nov 03 00:48:50.607472 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.607437 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-sys\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.607743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.607484 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmb8v\" (UniqueName: \"kubernetes.io/projected/e7553283-0a28-4c2f-9e4d-b742d467909b-kube-api-access-gmb8v\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.607743 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.607595 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-podres\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.607943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.607757 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-lib-modules\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.607943 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.607823 4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-proc\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709351 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709263 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-lib-modules\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709351 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709326 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-proc\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709378 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-sys\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709408 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmb8v\" (UniqueName: \"kubernetes.io/projected/e7553283-0a28-4c2f-9e4d-b742d467909b-kube-api-access-gmb8v\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709409 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-lib-modules\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709460 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-proc\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709467 4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-podres\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709510 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-podres\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.709590 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.709468 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7553283-0a28-4c2f-9e4d-b742d467909b-sys\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:50.771798 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:50.771765 4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmb8v\" (UniqueName: \"kubernetes.io/projected/e7553283-0a28-4c2f-9e4d-b742d467909b-kube-api-access-gmb8v\") pod \"perf-node-gather-daemonset-t5q56\" (UID: \"e7553283-0a28-4c2f-9e4d-b742d467909b\") " pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:51.051856 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.051759 4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:51.433847 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.433822 4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56"] Nov 03 00:48:51.435876 ip-10-0-135-179 kubenswrapper[4494]: W1103 00:48:51.435847 4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7553283_0a28_4c2f_9e4d_b742d467909b.slice/crio-ef19775a335a04a5610fe1fd1f284921061fa486fc74b8fb8a38d46f75520efa WatchSource:0}: Error finding container ef19775a335a04a5610fe1fd1f284921061fa486fc74b8fb8a38d46f75520efa: Status 404 returned error can't find the container with id ef19775a335a04a5610fe1fd1f284921061fa486fc74b8fb8a38d46f75520efa Nov 03 00:48:51.497555 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.497531 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-vhchr_ec7af7fd-654f-45db-a758-9307fb08247e/manager/0.log" Nov 03 00:48:51.585430 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.585399 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" event={"ID":"e7553283-0a28-4c2f-9e4d-b742d467909b","Type":"ContainerStarted","Data":"816c836e424a3933be19438a595d2341155f299c1972e5f87f23e764e85cf84c"} Nov 03 00:48:51.585612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.585453 4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" event={"ID":"e7553283-0a28-4c2f-9e4d-b742d467909b","Type":"ContainerStarted","Data":"ef19775a335a04a5610fe1fd1f284921061fa486fc74b8fb8a38d46f75520efa"} Nov 03 00:48:51.585612 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.585560 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:51.616801 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.616777 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8dkp_5d0a16e7-cd0c-4161-80ea-cdb46ae08b91/limitador/0.log" Nov 03 00:48:51.703536 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.703425 4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" podStartSLOduration=1.7034104719999998 podStartE2EDuration="1.703410472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:51.701922488 +0000 UTC m=+7246.131792346" watchObservedRunningTime="2026-11-03 00:48:51.703410472 +0000 UTC m=+7246.133280333" Nov 03 00:48:51.715633 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:51.715597 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:54.856006 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:54.855975 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/alertmanager/0.log" Nov 03 00:48:55.322102 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:55.322066 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/config-reloader/0.log" Nov 03 00:48:55.738976 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:55.738949 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/kube-rbac-proxy-web/0.log" Nov 03 00:48:56.153336 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:56.153258 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/kube-rbac-proxy/0.log" Nov 03 00:48:56.567533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:56.567501 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/kube-rbac-proxy-metric/0.log" Nov 03 00:48:56.892605 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:56.892520 4494 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-792jn_a686ab8e-a865-4f13-853b-3d43a3eef118/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:56.980164 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:56.980132 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/prom-label-proxy/0.log" Nov 03 00:48:57.394737 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:57.394701 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_26470e00-2773-4d56-b168-b3ce323c57df/init-config-reloader/0.log" Nov 03 00:48:57.603911 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:57.603884 4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6hpv/perf-node-gather-daemonset-t5q56" Nov 03 00:48:57.855542 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:57.855510 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7457749d7d-vhchr_ec7af7fd-654f-45db-a758-9307fb08247e/manager/0.log" Nov 03 00:48:59.484874 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:59.484842 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9f59dcf8c-bxllj_c9613f21-6e36-47cc-9633-c2865cd76577/metrics-server/0.log" Nov 03 00:48:59.907974 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:48:59.907904 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-w6tts_f1e6242e-8838-4cc2-b1ef-62e1a3f78bb9/monitoring-plugin/0.log" Nov 03 00:49:02.838197 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:02.838132 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtgpq_d1cb7b72-9689-4ce8-b206-9c39c0c2681c/node-exporter/0.log" Nov 03 00:49:03.252118 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:03.252085 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtgpq_d1cb7b72-9689-4ce8-b206-9c39c0c2681c/kube-rbac-proxy/0.log" Nov 03 00:49:03.669533 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:03.669509 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtgpq_d1cb7b72-9689-4ce8-b206-9c39c0c2681c/init-textfile/0.log" Nov 03 00:49:04.216384 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:04.215291 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hgxkp_e281f7d9-8170-49b5-a25a-891bbfa55133/kube-rbac-proxy-main/0.log" Nov 03 00:49:04.627200 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:04.627099 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hgxkp_e281f7d9-8170-49b5-a25a-891bbfa55133/kube-rbac-proxy-self/0.log" Nov 03 00:49:05.047034 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.046997 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hgxkp_e281f7d9-8170-49b5-a25a-891bbfa55133/openshift-state-metrics/0.log" Nov 03 00:49:05.346153 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.346067 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2645m_039a073e-0b49-4ab9-b6f6-79ece1993578/discovery/0.log" Nov 03 00:49:05.471969 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.471942 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/prometheus/0.log" Nov 03 00:49:05.665230 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.665204 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dbc8865f-r6rbc_edb2e3a6-f7f3-4333-b74e-7ebeceaf70ca/kube-auth-proxy/0.log" Nov 03 00:49:05.881719 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.881686 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-6wkzh_bf94d8f2-7b5b-4f04-88d7-2639a4ee6b4c/istio-proxy/0.log" Nov 03 00:49:05.884157 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:05.884135 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/config-reloader/0.log" Nov 03 00:49:06.305225 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:06.305199 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/thanos-sidecar/0.log" Nov 03 00:49:06.316579 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:06.316557 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8fv4v_0a1eab1e-ff5b-4337-8058-c10970f7761e/istio-proxy/0.log" Nov 03 00:49:06.720300 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:06.720253 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/kube-rbac-proxy-web/0.log" Nov 03 00:49:06.858786 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:06.858758 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-k2dss_80b38964-2c95-47b9-bea3-6a2d26f224cc/istio-proxy/0.log" Nov 03 00:49:07.133689 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:07.133612 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/kube-rbac-proxy/0.log" Nov 03 00:49:07.496785 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:07.496756 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-m8ss5_ec3b4496-e1c1-464c-bbfb-d290f8815c66/istio-proxy/0.log" Nov 03 00:49:07.547591 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:07.547568 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/kube-rbac-proxy-thanos/0.log" Nov 03 00:49:07.964493 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:07.964466 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545d7ff-bdcb-40bd-981c-d6bfd5320434/init-config-reloader/0.log" Nov 03 00:49:09.644767 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:09.644738 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f595c5795-zfd8j_866403c0-bfc0-4e67-8424-1686b798b6ba/telemeter-client/0.log" Nov 03 00:49:10.061232 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:10.061207 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f595c5795-zfd8j_866403c0-bfc0-4e67-8424-1686b798b6ba/reload/0.log" Nov 03 00:49:10.474931 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:10.474899 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f595c5795-zfd8j_866403c0-bfc0-4e67-8424-1686b798b6ba/kube-rbac-proxy/0.log" Nov 03 00:49:10.900201 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:10.900093 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/thanos-query/0.log" Nov 03 00:49:11.319906 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:11.319877 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/kube-rbac-proxy-web/0.log" Nov 03 00:49:11.735391 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:11.735359 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/kube-rbac-proxy/0.log" Nov 03 00:49:12.148396 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:12.148305 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/prom-label-proxy/0.log" Nov 03 00:49:12.562625 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:12.562600 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/kube-rbac-proxy-rules/0.log" Nov 03 00:49:12.975888 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:12.975856 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb855d47f-vn7b2_7939c857-e771-4360-a44b-0b0113a23841/kube-rbac-proxy-metrics/0.log" Nov 03 00:49:15.053252 ip-10-0-135-179 kubenswrapper[4494]: I1103 00:49:15.053222 4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8dkp_5d0a16e7-cd0c-4161-80ea-cdb46ae08b91/limitador/0.log"